View
217
Download
0
Tags:
Embed Size (px)
Citation preview
11
Defending Distributed Systems Defending Distributed Systems Against Malicious Intrusions and Against Malicious Intrusions and Network AnomaliesNetwork Anomalies
Kai HwangKai HwangInternet and Grid Computing Laboratory Internet and Grid Computing Laboratory University of Southern CaliforniaUniversity of Southern California
Keynote PresentationKeynote Presentationat theat the IEEE International Workshop on Security in Systems and Networks IEEE International Workshop on Security in Systems and Networks (SSN-2005),(SSN-2005), held in conjunction withheld in conjunction with thethe IEEE International Parallel and Distributed Processing IEEE International Parallel and Distributed Processing Symposium Symposium (IPDPS-2005),(IPDPS-2005), Denver, Colorado, April 8, 2005Denver, Colorado, April 8, 2005
This presentation is based on research findings by USC GridSec team. Project Web site: This presentation is based on research findings by USC GridSec team. Project Web site:
http://GridSec.usc.eduhttp://GridSec.usc.edu,, supported by NSF ITR Grant No. 0325409, and contributed by supported by NSF ITR Grant No. 0325409, and contributed by
Min Cai, Shanshan Song, Ricky Kwok, Ying Chen, and Hua LiuMin Cai, Shanshan Song, Ricky Kwok, Ying Chen, and Hua Liu
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 22
Presentation Outline:Presentation Outline:
Security/privacy demands in networked Security/privacy demands in networked
or distributed computer systemsor distributed computer systems
GridSec NetShield architecture for defending GridSec NetShield architecture for defending
distributed resource sites in Grids, clusters, etc.distributed resource sites in Grids, clusters, etc.
Internet datamining for collaborative anomaly and Internet datamining for collaborative anomaly and
intrusion detection system (CAIDS) with traffic intrusion detection system (CAIDS) with traffic
episode rule training and analysisepisode rule training and analysis
Fast containment of internet worm outbreaks and Fast containment of internet worm outbreaks and
tracking of related DDoS attacks with distributed-tracking of related DDoS attacks with distributed-
hashing overlays hashing overlays
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 33
Security and Privacy Demands inSecurity and Privacy Demands inNetwork and Distributed SystemsNetwork and Distributed Systems
Trusted resource allocation, sharing, and schedulingTrusted resource allocation, sharing, and scheduling
Secure communications among resource sites, clusters, Secure communications among resource sites, clusters, and protected download among peer machinesand protected download among peer machines
Intrusion and anomaly detection, attack repelling, trace Intrusion and anomaly detection, attack repelling, trace back, pushback of attacks, etcback, pushback of attacks, etc
Fortification of hardware/software (firewalls, packet filters, Fortification of hardware/software (firewalls, packet filters,
VPN gateways, traffic monitors, security overlays, etc. )VPN gateways, traffic monitors, security overlays, etc. )
Self-defense toolkits/middleware for distributed defense, Self-defense toolkits/middleware for distributed defense, risk assessment, worm containment, response automationrisk assessment, worm containment, response automation
Anonymity, confidentiality, data integrity, fine- grain access Anonymity, confidentiality, data integrity, fine- grain access control, resolving conflicts in security policies, etc control, resolving conflicts in security policies, etc
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 44
GridSecGridSec: A Grid Security ITR Project at USC: A Grid Security ITR Project at USC
Steps for automated self-defense at resource site : Step 1: Intrusion detected by host-based firewall /IDS Step 2: All VPN gateways are alerted with the intrusions Step 3: Gateways broadcast response commands to all hosts
VPN Gateway
3
3
3
Site S1
3
3Host
1
2
Internet
Host
Host
3
3
Site S2 Site S3
2
3
3
3
VPN Gateway VPN
Gateway
Host
Host
Host
Host
Host
Host
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 55
The NetShield Architecture with Distributed The NetShield Architecture with Distributed Security Enforcement over a DHT OverlaySecurity Enforcement over a DHT Overlay
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 66
Building Encrypted Tunnels between Grid Building Encrypted Tunnels between Grid Resource Sites Through the DHT OverlayResource Sites Through the DHT Overlay
The number of encrypted tunnels should grow with O(N) The number of encrypted tunnels should grow with O(N)
instead of O(N x N), where N is the number of Grid sites instead of O(N x N), where N is the number of Grid sites
Using shortest path, security policy is enforced Using shortest path, security policy is enforced
with minimal VPN tunnels to satisfy special Grid with minimal VPN tunnels to satisfy special Grid
requirements, automaticallyrequirements, automatically
How to integrate security policies from various private How to integrate security policies from various private
networks through the public network ?networks through the public network ?
How to resolve security policy conflicts among hosts, How to resolve security policy conflicts among hosts,
firewalls, switches, routers, and servers, etc. in a Grid firewalls, switches, routers, and servers, etc. in a Grid
environment ?environment ?
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 77
Trust IntegrationTrust Integration over a DHT Overlayover a DHT Overlay
Cooperating gateways working together to establish VPN tunnels for trust integration
Physical backbone
DHT Overlay Ring
Trust Vector
Trust vector propagation
User application and SeGO server negotiation
V
SeGO Server Hosts
VPN Gateway
Site S3
Site S2
Site S1
Site S4
V
V
V
V
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 88
USC NetShield Intrusion Defense System USC NetShield Intrusion Defense System
for Protecting Local Network of for Protecting Local Network of Grid Computing ResourcesGrid Computing Resources
Network
Router
The Internet
The Internet
ISP
The NetShield System
Victim’s Internal
Network
Datamining for Anomaly Intrusion Detection (IDS) Firewall
Risk Assessment System (RAS)
Intrusion Response
System (IRS)
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 99
Alert Operations performed in local Grid Alert Operations performed in local Grid sites and correlated globallysites and correlated globally
Local alert correlation Global alert correlation
IDS IDS IDS
Alert classification
DHT module Global alert clustering
Alert merging
Alert Assessment Reporting, and Reaction
Alerts Local alert clustering
Intrusion reports
Alert correlation Alert clusters
Alert formatting
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 1010
Basic Concept of Internet EpisodesBasic Concept of Internet Episodes
Event Type:Event Type: A, B, C, D, E, F, etc. A, B, C, D, E, F, etc.
Event Sequence:Event Sequence: e.g., <(E,31),(D,32),(F,33)> e.g., <(E,31),(D,32),(F,33)>
Window:Window: Event sequence with a particular width Event sequence with a particular width
Episode:Episode: partially ordered set of events, e.g. whenever A occurs, B partially ordered set of events, e.g. whenever A occurs, B will occur soonwill occur soon
Frequency of episode:Frequency of episode: fraction of windows in which episode occurs fraction of windows in which episode occurs
Frequent episode:Frequent episode: set of episodes having a frequency over a set of episodes having a frequency over a particular frequency thresholdparticular frequency threshold
Frequent episode rulesFrequent episode rules are generated to describe the are generated to describe the connection eventsconnection events
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 1111
Frequent Episode Rules (FER) Frequent Episode Rules (FER) for Characterizingfor Characterizing Network Traffic ConnectionsNetwork Traffic Connections
E → D, F ( c, s )The episode of 3 connection events (E, D, F) = (http, smtp, telnet).
On the LHS , we have the earlier event E (http). On the RHS, we have
two consequence events D (smtp) and F(telnet); where s is the
support probablity and c is the confidence level specified below:
(service = http, flag = SF) →
(service = smtp, srcbyte = 5000),
(service = telnet, flag = SF) (0.8, 0.9)
Support probability s = 0.9 and Confidence level c = 0.8 that the
episode will take place in a typical traffic stream
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 1212
AA Cooperative Anomaly and Intrusion Detection System Cooperative Anomaly and Intrusion Detection System (CAIDS), (CAIDS), built with abuilt with a Network Intrusion Detection System Network Intrusion Detection System
(NIDS) (NIDS) and anand an Anomaly Detection System (ADS) Anomaly Detection System (ADS) operating operating interactively through automated signature generationinteractively through automated signature generation
ADS
Episode Mining Engine
Known attack signatures from ISD provider
IDSSignature Matching
Engine
Attack Signature Database
Episode Rule Database
Signature Generator
Audit records from traffic data
Single-connection attacks detected at packet level
Training data from audit normal traffic
records
Anomalies detected over multiple connections
New signaturesfrom anomalies detected
Unknown or burst attacks
ADSADS
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 1313
Internet Datamining for Episode Rule Generation
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 1414
Attack Spectrum Attack Spectrum from MIT Lincoln from MIT Lincoln LabLab in 10 Days of Experimentationin 10 Days of Experimentation
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 1515
Automated Signature Generation from Automated Signature Generation from Frequent Episode AnalysisFrequent Episode Analysis
1. Label relevant connections toassociate with an FER.
Episode rules matching the normal
FER database ?
2 Check error flags or other useful temporal statistics
3 Extract common features suchas IP addresses, protocol, etc.to form the signature
Episode Frequency exceeding the rule
threshold ?
Yes
2 Calculate additional information such as connection count, average and percentage of connections, etc.
3 Select one of the predefined classifiers 4 Use the selected classifier to classify the attack class
and find the relevant connections5 Extract common features in all identified
connections, such as the IP addresses, protocol, etc. to form the signature
Adding new signatures to the Snort database
Ignore the normal episode rules from legitimate users (No anomaly detected)
No
No (Stealthy attacks)
Online traffic episode rules from the datamining engine
Yes (Massive attacks)
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 1616
Successful Detection Rates of Snort , Anomaly Detection System (ADS), and the Collaborative Anomaly and Intrusion Detection System (CAIDS)
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 1717
0
2
4
6
8
10
12
14
16
18
100 300 500 1000 7200
Wi ndow Si ze (Second)
Numb
er o
f Fa
lse
Alar
ms
R2LDoSProbeU2R
False Alarms out of 201 Attacks in CAIDS Triggered by Different Attack Types
under Various Scanning Window Sizes
Using larger windows result in more false alarms. Shorter windows in 300 sec or less are better in the sense that shorter episodes will be mined to produce shorter rules, leading to faster rule matching in the anomaly detection process
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 1818
Detection Rates of Snort, ADS, and CAIDSunder Various Attack Classes
On the average, the CAIDS (white bars) outperforms
the Snort and ADS by 51% and 40%, respectively
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 1919
ROC Curves for 4 Attack Classes ROC Curves for 4 Attack Classes on The Simulated CAIDSon The Simulated CAIDS
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 2020
ROC Performance of Three ROC Performance of Three Intrusion Detection SystemsIntrusion Detection Systems
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 2121
Internet Worm and Flood Control:Internet Worm and Flood Control:
A DHT-based WormShield overlay network is under A DHT-based WormShield overlay network is under
development at USC. development at USC.
Fast worm signature generation and fast Fast worm signature generation and fast
dissemination through both local and global dissemination through both local and global
address dispersionaddress dispersion
Automated tracking of DDoS attack-transit routers Automated tracking of DDoS attack-transit routers
to cut off malicious packet flows for dynamic DDoS to cut off malicious packet flows for dynamic DDoS
flood control flood control
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 2222
The WormShield Built with a DHT-basedThe WormShield Built with a DHT-based Overlay with Six Worm Monitors Overlay with Six Worm Monitors
Chord IDContentBlock
LocalPrevelance
(src, dest)Addresses
76 s1 1 S1(A), D1(A)
112 s2 4 S2(A), D2(A)
55 s3 2 S3(A), D3(A)
215 s4 5 S4(A),D4(A)
Site A
Site B
Site C
Site D
Site F
0/256
192
128
64
Site E
Chord IDContentBlock
GlobalPrevelance
AddressDispersion
215 s4 5+6=11 18180 s5 4+8=12 22...
3lTLocal Table:
Global Table: 2010 cp T,T
Chord IDContentBlock
LocalPrevelance
(src, dest)Addresses
215 s4 6 S4(C),D4(C)
180 s5 4 S5(C),D5(C)
3lTLocal Table:Chord ID
ContentBlock
LocalPrevelance
(src, dest)Addresses
180 s5 7 S5(D),D5(D)
3lTLocal Table:
IdentifiedWorm
Signature!
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 2323
The WormShield Signature Generation ProcessThe WormShield Signature Generation Process
Chord Protocol
OtherWormShield
Monitors
Monitored DMZ Traffic
Loca
l Co
nten
t Pre
vale
nce
Tab
le
Chord ID ContentBlock
LocalPrevelance
ID(j) j L(i, j)
Content Block j
L(i, j)> Tl
Rabin Fingerprinting
Update L(i,j)
Loca
l Add
ress
Dis
pers
ion
Ta
ble
ContentBlock SRC IP DEST IP
j S(i, j) D(i, j)Update
S(i,j), D(i, j)
|S(i,j)|+|D(i,j)|> Ts
Send updatesfor P(j) and C(j) to
monitor root(j)
Glo
bal C
onte
nt P
reva
lenc
e &
Add
ress
Dis
pers
ion
Tab
le
Chord ID GlobalPrevelance
ID(j) P( j)
AddressDispersion
C(j)
P(j) > Tp&& C(j) > Tc
No
Yes
No
Yes
No
Update P(j), C(j)
Yes
Process updatesfor P(j) and C(j) from
other monitors
Report j assuspected worm
Disseminate suspectedworm signature j to
WormShield network
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 2424
Signature Detection in Worm Spreading and the Signature Detection in Worm Spreading and the Growth of Infected hosts for Simulated CodeRed Growth of Infected hosts for Simulated CodeRed
Worms on a Internet Configuration of 105,246 Edge Worms on a Internet Configuration of 105,246 Edge networks in 11,342 Autonomous Systems networks in 11,342 Autonomous Systems
Containing 338,652 Vulnerable HostsContaining 338,652 Vulnerable Hosts
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 2525
Effects of Local Prevalence ThresholdEffects of Local Prevalence ThresholdWorm spreading and the growth of infected hostsWorm spreading and the growth of infected hosts
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 2626
Effects of Global Address PrevalenceEffects of Global Address Prevalence on on Worm Spreading and the Growth of Infected HostsWorm Spreading and the Growth of Infected Hosts
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 2727
Reduction of Infected Hosts by Reduction of Infected Hosts by Independent vs. Collaborative Independent vs. Collaborative
Monitoring over the Edge NetworksMonitoring over the Edge Networks
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 2828
Packet/Flow Counting for Tracking Packet/Flow Counting for Tracking Attack-Transit Routers (ATRs)Attack-Transit Routers (ATRs)
IngressRouter
Last Hop Router
Victim
IngressRouter
IngressRouter
Attack FlowsAttack Flows
Legitimate Flows
Legi
timat
e Fl
ow
Legi
timat
e Fl
ow
Tracking andFlood Control
Identifiedas an ATR
Packet-level Traffic Matrix A
Flow-level Traffic Matrix B
Identifiedas an ATR
LogLogCardinality Summary
LogLogCardinality Summary
LogLogCardinality Summary
LogLogCardinality Summary
LogLogCardinality Summary
LogLogCardinality Summary
Tracking andFlood Control
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 2929
False Positive Rate of Identified ATRsFalse Positive Rate of Identified ATRs
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 3030
Other Hot Security Research Areas:Other Hot Security Research Areas: Efficient and enforceable trust models are very much in Efficient and enforceable trust models are very much in
demand for networked and distributed systems: PKI demand for networked and distributed systems: PKI
services, VPN tunneling, trust negotiation, security overlays, services, VPN tunneling, trust negotiation, security overlays,
reputation system etc.reputation system etc.
Large-scale security benchmark experiments in open Internet Large-scale security benchmark experiments in open Internet
environments are infeasible. The NSF/HSD DETER testbed environments are infeasible. The NSF/HSD DETER testbed
should be fully used in performing such experiments to should be fully used in performing such experiments to
establish sustainable cybertrust over all edge networks. establish sustainable cybertrust over all edge networks.
Internet datamining for security control and for the guarantee Internet datamining for security control and for the guarantee
of Quality-of-Service in real-life network applications – of Quality-of-Service in real-life network applications –
Interoperability between wired and wireless networks is Interoperability between wired and wireless networks is
a wide-open area for further research.a wide-open area for further research.
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 3131
Final RemarksFinal Remarks The NetShield built with DHT-based security overlay
networks support distributed intrusion and anomaly detection, alert correlation, collaborative worm containment, and flooding attack suppression.
The CAIDS can cope with both known and unknown network attacks, secure many cluster/Grid/P2P operations in using common Internet services: telnet, http, ftp, Email, SMTP, authentication, etc.
Automated virus or worm signature generation plays a vital role to monitory network epidemic outbreaks and to give early warning of large-scale system intrusions, network anomalies, and DDoS flood attacks. Extensive benchmark experiments on the DETER test bed will prove the effectiveness.
http://GridSec.usc.eduhttp://GridSec.usc.eduApril 8, 2005, Kai HwangApril 8, 2005, Kai Hwang 3232
Recent Related Papers:Recent Related Papers: 1.1. M. Cai, K. Hwang, Y. K. Kwok, Y. Chen, and S. S. Song, “Fast M. Cai, K. Hwang, Y. K. Kwok, Y. Chen, and S. S. Song, “Fast
Containment of Internet Worms and Tracking of DDoS Attacks with Containment of Internet Worms and Tracking of DDoS Attacks with Distributed-Hashing Overlays”, Distributed-Hashing Overlays”, IEEE Security and Privacy,IEEE Security and Privacy, accepted accepted to appear Nov/Dec. 2005.to appear Nov/Dec. 2005.
2.2. K. Hwang, Y. Kwok, S. Song, M. Cai, R. Zhou, Yu. Chen, Ying. Chen, K. Hwang, Y. Kwok, S. Song, M. Cai, R. Zhou, Yu. Chen, Ying. Chen, and X. Lou, “GridSec: Trusted Grid Computing with Security Binding and X. Lou, “GridSec: Trusted Grid Computing with Security Binding and Self-Defense against Network Worms and DDoS Attacks”, and Self-Defense against Network Worms and DDoS Attacks”, International Workshop on Grid Computing Security and Resource International Workshop on Grid Computing Security and Resource ManagementManagement (GSRM’05), (GSRM’05), in conjunction with in conjunction with ICCS 2005ICCS 2005, Atlanta, May , Atlanta, May 22-25, 2005.22-25, 2005.
3.3. M. Qin and K. Hwang, “Frequent Episode Rules for Internet Traffic M. Qin and K. Hwang, “Frequent Episode Rules for Internet Traffic Analysis and Anomaly Detection”, Analysis and Anomaly Detection”, IEEE Network Computing and IEEE Network Computing and Application Symp. Application Symp. ((NCA-2004NCA-2004),), Cambridge, MA. August 31, 2004 Cambridge, MA. August 31, 2004
4.4. K. Hwang, Y. Chen and H. Liu, “ Defending Distributed Computing K. Hwang, Y. Chen and H. Liu, “ Defending Distributed Computing Systems from Malicious Intrusions and Network Anomalies”, Systems from Malicious Intrusions and Network Anomalies”, IEEE IEEE Workshop on Security in Systems and NetworksWorkshop on Security in Systems and Networks (SSN’05), (SSN’05), in in conjunction with IEEE conjunction with IEEE IPDPS 2005IPDPS 2005, Denver, April 8, 2005. , Denver, April 8, 2005.