Application-IntegratedData Collectionfor Security Monitoring

Magnus Almgren and Ulf LindqvistSystem Design Lab, SRI International

The work described here was funded by DARPA under contract number F30602-99-C-0149 and F30602-98-C-0059. The views herein are those of the authors and do not necessarily reflect the views of the supporting agency.

2

IntroductionHow many network IDSs can detect this?

How many network IDSs can detect this?

3

Outline Part OneTraditional NIDS and HIDSHost-located data collectionApplication-integrated moduleProof-of-concept implementationImplementation architecture and data flow

Part Two

4

Traditional Network IDSProblem areas:

Encrypted trafficEvasion tricksNetwork speedSession/transaction reconstruction and statefulnessTimely preemption difficult

Advantages:Passive, non-invasiveHiddenCan monitor multiple hosts from one location

5

Traditional Host IDSData sources

Audit data – useful, but limited insight into application dataApplication/system log files – limited content, disk space management

Usually, data produced after the factBlind to most network-level attacks

6

Host-located Real-time Event Data Collection on Multiple Levels

Application-integrated IDSApplication

OS IDS analyzing audit trail (system calls)

Network Network IDS (located on network or host)

Security violations manifest themselves differently on different levelsThe data sources are complementary

7

Application-integrated ModuleAdvantages

Unencrypted informationIndependent of network speedDetailed information availableTrue session and transaction decoding and reconstructionOpportunity for preemptive capability

DisadvantagesTailored for a specific applicationInvasive: Could impact application performance and stability

Complements data collection on other levels

8

Proof-of-concept: One type of application

Web serverWeb is a popular, ubiquitous serviceAllowed through most firewallsExisting EMERALD analysis engine for HTTP Many Web servers allow custom extensions

Apache Web serverApache ~60% of market according to NetcraftOpen-source, well-documented module interface

9

Host

Web server

Implementation Architecture and Data Flow

1. The Web server receives a request

2. Module produces transaction data

3. Message is sent to eXpert-HTTP

4. eXpert-HTTP performs analysiseXpert-HTTP

Datacollectionmodule

EMERALDlibraries

Transaction recordsas EMERALD messages

Alerts

10

Outline Part TwoInside the Apache ServerPerformance evaluationEvasion techniquesProblems with this approachFuture workRelated approachesConclusions

11

Inside the Apache ServerApache uses a request loop Hooks are available in all phases of the loopOur module is hooked to the logging phase

Currently no feedback from analysis unitPassive data collection

[Stein 1999]

12

Performance Experiment SetupGoal: Measure impact on user experienceUsed WebLoad from RadView Software

Set up to request a single URL repeatedlyMeasured transaction round-trip timeEach run was 60 minutes with 10 virtual clients on a single physical host

Static page: text and one imageDynamic page: CGI program

13

Performance Results: Static Page50 KB text + 12 KB JPEG

0.0590.057std dev1.5%1.5211.499average2.1%1.5171.486medianImpactW/ IDSNo IDSRound-trip (s)

14

Performance Results: Dynamic PageExecution of a CGI program

0.0480.034std dev3.6%1.2381.195average3.1%1.2291.192medianImpactW/ IDSNo IDSRound-trip (s)

15

Evasion TechniquesUsing lower protocol levels [Ptacek and Newsham]

Crafting ambiguous HTTP request

1. GET /cgi-bin/phf

2. GET /%00cgi-bin/phf

3. GET / HTTP 1.1Host: victimContent-Length: 3

123GET /cgi-bin/phf

The evasion techniques work because Web servers and NIDS decode them differently

Tab

16

Problems1) Invasive

Because the module must run within the server application, it could impact stabilityTesting is difficult

Server applications typically do not run from batch inputForced to use scripts rather than data files

2) Application-specificA module must be written specifically for every application you want to monitorEvery application has its own interface (or none) for customized moduleBut: For many services, only a few major brands

17

Future WorkOther Web server brands

iPlanet – prototype ready

Other servicesFTPSMTP (sendmail)Remote access (telnet, rlogin)Databases

Great potential for improved analysis

In many cases, knows how the request was serviced (e g document or CGI program)Could detect evasion attemptsKnows the exact local filename request refers toCheck expected control flow of program

18

Preemption: Two-tiered approachProblem:

Complete analysis in module ⇒ performance hitAnalysis on separate host ⇒ excludes preemption

Solution: Two-tier analysisModule contains simple (fast) analysis engine: hash of suspicious source/CGI programeXpert-HTTP performs complete (slower) analysis and keeps a global stateeXpert updates the module’s (simple) knowledgeThree options for server module: serve request, deny request, or wait for further analysis

19

Per-request GranularitySingle requests can be stopped or delayed (awaiting analysis)Select requests for analysis depending on the type (static, dynamic, directory)Compare with a packet-filtering firewall

IP address/port granularityCan only block, not delay

20

Related Approachesmod_id by Burak Dayioglu (www.dayioglu.net)

Performs simple CGI name matching inside module (development discontinued)

TripWire for Web pagesLimited to stopping altered content from being served (MD5 checksums)

Interfacing Trusted Applications with Intrusion Detection Systems Marc Welz and Andrew Hutchison, RAID 2001

21

ConclusionsApplication-integrated data collection for IDS complements data from other levels and locationsAddresses the three most severe problems NIDS are currently facing:

EncryptionEvasionNetwork speed

Prototype integrated with Apache and with EMERALD infrastructure and analysis engine