Intersect

© 2011 The MITRE Corporation. ALL RIGHTS RESERVED.Approved for Public Release: 11-0130

INTERSECT: Combining Commercial/FOSS Tools with Custom Code to Root Out

Malware

Fotios LindiakosMatt Pawloski


INTERSECT: How to Combine All of the Stuff You Spent Too Much Money on With

the Cool Free Stuff Your Boss Won’t Let You Install to Actually Do Something

Useful



About Us

Fotios• Graduated 2007 from

RIT with a BS in CS• Attending GMU for a

MS in ISA• Started as an intern at

MITRE and has been full time for 3.5 years

Matt• Graduated 2005 from

RIT with a BS in IT• Graduated 2010 from

Capitol College with MS in IA

• Worked at Symantec, KCG

• Been at MITRE 3 years


About MITRE

• “Not-for-profit organization chartered to work in the public interest”

• “MITRE is a unique organization that assists the United States government with scientific research and analysis, development and acquisition, and/or systems engineering and integration”

• “MITRE also has its own independent research and development program that explores new technologies and new uses of technologies to solve our sponsors' problems in the near-term and in the future”

Sources: http://www.mitre.org/about/ffrdcs.htmlhttp://www.mitre.org/about/index.html

http://www.mitre.org/about/ffrdcs.html

http://www.mitre.org/about/ffrdcs.html

http://www.mitre.org/about/index.html


An Axiom

• Client side attacks are the most prevalent attack vector– Users receiving a malicious email attachment– Users receiving a malicious link in an email

• We need agile file examination!• Good tools exist, but can be hard to

deploy/use• “Real-time” is nice to have, but not

practical


STATE OF THE INDUSTRY


ProductsPros Cons

Antivirus• Quick

indicator• Cheap/free

• Not effective against targeted attacks

IDS/IPS

• Can block in real-time

• Same “signature problem” as antivirus

• Doesn’t examine full files• False positives can cause an

outage

Home Grown

• Can be very effective for your specific organization if used properly

• Getting files can be difficult• Lots of reinvention of the

wheel• Can be unstable


Most COTS Products

• Difficult to interface with• No great “top to bottom” solution• Expensive• Not agile enough to meet quickly

adapting threats• Vendors don’t meet your specific

needs• This doesn’t mean they are

worthless!


If DEFCON has taught us anything…

• “Race to Zero”– Signature based scanning is trivial to bypass– Examples

• Repacking• Causing AV engines to timeout by wrapping

malware with some trivial code– Doesn’t have to attack AV or modify malware

• Unhook AV

• Targeted defenses are needed for targeted attacks


CURRENT THREATS


Client Side Attacks

• We have everything we need in the file– Static analysis• Initial file is usually just a dropper

– Behavioral analysis• File will beacon out, download more

malware, and commence C&C

• Even a small success rate is still a success


0-day

• Everybody is sick of talking about this

• Detection sometimes possible through– Content detonation– Targeted profiling• ssdeep• HBGary FingerPrint


Targeted Attacks

• How can you expect any non-specialized tools to find something that was made specifically for you?– Targeted attacks need targeted defenses

• Only targeted at a select number of users

• Phishing email so well crafted your users will definitely click on it


What We Need

• Need to Gather Intelligence– Accumulating Data

• You already have the files• Different tools can tell you different things

– Correlating Data

• Need to Protect– Detect targeted attacks– Need to react faster than traditional solutions– Different tools may offer overlapping protection

• Need to Measure Efficacy– Are your tools actually doing a good job?– Easily evaluate new technologies


WHAT WE PROPOSE


Not a Replacement!

• You still need these products• We want to augment and integrate

them• Use conventional technology in

unconventional ways


Our Requirements

• Simple Interfaces– Front end (user experience)– Back end (developer experience)

• Scalable• Resilient• Fast• Awesome (with a catchy name)


Our Solution…

• We call it INTERSECT


What Is It?

• Middleware that works!• Just a framework– Ties together all the pieces (more on

them later)– Gives the users a “single pane of glass”– Handles all of the mundane stuff to let

the developers focus on their parts– Helps consolidate results• Can be used to perform correlation and

alerting


Producers

• Any services/devices that see full files and can upload

• Examples–Web proxy– Email server– File server (SMB, FTP, etc)– Full file extractor on live network stream


Producer Example

#!/usr/bin/rubyrequire ‘rest_client’

RestClient.post url, {:upload => {

:upload => File.new(filename)},:transfer => {

:param1 => value1,:param2 => value2

}}


Consumers

• Scanners that examine or possibly modify files submitted by producers– Start from scratch• Code it right into your own tools

– Leverage existing tools• Write a wrapper for a COTS product you

already have

• Return results to be correlated


Examples of Consumers

COTS• AV

– Individual (Symantec, McAfee, etc)

– Aggregate (MetaScan)

• Content Detonation– FireEye

• File Profiling– HBGary FingerPrint

FOSS/Custom• AV

– Individual (ClamAV, AVG, etc)– Aggregate (VirusTotal)

• Content Detonation– Honeynet Project

• Yara– Public signatures– Create your own!

• Archive Extraction– Zip/Tar/ISO– DD Image. Forensics anybody?

• Covert Data Channels– Find indicators and quickly

weaponize them


Consumer Exampledef subscribe()

AMQP.start(:host => HOST ) doamq = MQ.newq = amq.queue(QUEUE_NAME)ex = amq.topic(MESSAGES_EX)

q.bind(ex, :key => "image.#")

q.subscribe(:ack => true) do |hdr,body|

yield hdr,bodyhdr.ack

endend

end


Returning Results

def publish(hash)AMQP.start(:host => HOST ) do

amq = MQ.newex = amq.fanout(RESULTS_EX)

ex.publish(hash[:body],:headers => hash[:headers]

)end

end


INTERSECT INTERNALS


Brains

• Keep the trains running…• Accepts file submissions• Submit files to bus• Collect results created by consumers• Correlate results from consumers


Front End

• HTTP Interface• Producers can upload files– By POSTing files

• Analysts can view results– Through the Ruby on Rails app

• Consumers can download files to analyze– Via a simple static file serving


Ruby on Rails App

• Allows for rapid web development– Abstracts away everything for you

• WEBrick– Standard, lightweight development web server– Holding up to pretty much whatever we throw at it

• EventMachine– Extremely high scalability, performance and stability

for the most demanding production environments– An API that eliminates the complexities of high-

performance threaded network programming, allowing engineers to concentrate on their application logic


Back End

• MySQL– Holds all metadata for files and results

• File store– All files are stored and renamed to

match their MD5 hash to prevent duplication


Service Bus

• The unsung hero• Disclaimer–We know very little about ESB software.

We know just enough to say we don’t like them.

• Provides basic routing of messages between producers, INTERSECT, and consumers

• Allows us to decouple everything– Just connect to the bus


Tying It Together

• Consumers bind to the bus– Simple wrapper script / class to

communicate with bus• Use this method to quickly repurpose

already existing services and capabilities• Use this method to integrate proprietary

solutions with limited interfaces

– Integrate directly into your consumer from the start


Advanced Message Queuing Protocol (AMQP)

• Awesome protocol– Lightweight

• Developed by some financial companies to facilitate “common business messaging”

• Protocol developed by some major technical companies


RabbitMQ

• Easy to setup– Servers run on Windows, *nix, OSX, OpenVMS?!

• Libraries for most languages– Ruby, Java, Perl, Python, .NET, PHP, C, Erlang,

Lisp, Haskell

• Simple to configure and manage– Allows you to spend more time developing– Powerful features

• Access control• vhosts• Load balancing / Redundancy


Exchanges

Fanout• Shotgun approach

– All services get all messages

– Queues fill up– Consumers have to

decide

Topic• More precise

– Declare what messages you want based on routing key

– We use filetype• Based on libmagic for

now

– Reduces load on consumers


Messages

• We keep them as small as possible• Files are not contained in the

messages– Some consumers simply operate on

metadata

• We provide a URL to get the file from INTERSECT


Message Packet Capture


Result Packet Capture


Consumer Workflow


Load Balancing / Redundancy

• RabbitMQ–Multiple instances can be stood up in a

cluster

• Consumer–Multiple instances can be bound to the

same queue–Messages will be delivered to an

available instance of a service


ADVANTAGES TO OUR APPROACH


“Single Pane of Glass”

• Consolidate results from disparate services– Correlate those results to find something

novel

• Search through results and transfers by any amount of metadata

• Evaluate efficacy of different services– Did some detect maliciousness and

others not?


Agile

• Easy to add new consumers• Resubmit files to new/updated

consumers• Provide research projects with

relevant test data


Asynchronous

• Queuing allows consumers to take as much time as they need to process files

• A failure of one consumer has no effect across the system– Less stable research projects can

process real data to better prove their methods


Leverage Resources

• Fully utilize your COTS tools– It was expensive, get your money’s

worth–Many of them expect a manual workflow

and go underutilized

• Throw more hardware at it– Run multiple copies of services


WHAT WE NEED


What We Need

• Need to Gather Intelligence• Need to Protect• Need to Measure Efficacy


Gather Intelligence

• You can never have too much intelligence

• Once you have all of the information in one place– Act on it– Analyze it


Protect

• How we’re using it right now• Producers –Web proxy– Network taps–Mail server

• Consumers– Lots of file scanners


Protect

• Workflow– File comes in from the network

• Somebody is downloading or was sent a file• Scanners do triage• If the file is suspicious

– Alert an analyst– They can decide what to do based on your

corporate policy

• We can accumulate data on files– Retroactively scan files when new tips/signatures

come out– Start to tie different files to the same attackers


Statistics

• Began ingesting files from multiple enterprise producers

• Since 14-Dec–~175,000 unique files– Averaging ~4500 unique files daily–Max ~7800 files in a day– No backlog!


Measure Efficacy

• With all of that data– You can see how your COTS tools

compare to your research projects– You can see how your research projects

are progressing• Run scans with one version• After you make some changes, run new

scans and compare


Other Possible Use Cases

• Egress Filtering–Write scanners that look for SSN, Credit

Card Numbers, “Dirty Words”…

• File Transfer–Make users put files into the system if

they want to bring it into the corporation– Don’t allow them to download it directly• They need to put it into your system and

then download it from there after if gets scanned


FUTURE IMPROVEMENTS


Correlation Engine Integration

• We are working on utilizing Splunk– Other SIEMs would work too

• Provide better UI, alerting, searching, etc.

• Don’t reinvent the wheel


Better Filetype Checking

• This is a difficult problem to get perfectly correct

• Maybe we can develop services that do this for us…

> file --mime-type INTERSECT.*INTERSECT.doc: application/mswordINTERSECT.docm: application/zipINTERSECT.docx: application/zip


Archive “Explosion”

• Simplest form, unzip a file and resubmit the children– The “threat” of the archive would be an aggregate

of the threat of the children

• But what is an “archive”?– Office 2007+ file format

• PowerPoint stores each slide as a different file, along with each image individually

– Disk images• Write a forensic service that can parse through and pull

out all files• Resubmit those files to keep track of all files in a disk

image


Contact Us

• Emails– [email protected]–[email protected]

mailto:[email protected]

mailto:[email protected]

Documents

Intersect