MOLOCH: Search for Full Packet Capture (OA Cyber Summit)

MOLOCH:SEARCH FOR FULL PACKET

CAPTURE

It is a Great Horned Owl

Project Logo

WHY THE OWL?

Owls are silent hunters that go after RATs. We think that’s pretty

cool.

3

HISTORYLESSONWHY AOL BUILT MOLOCH

WHAT IS MOLOCH?

5

Moloch is an open source, scalable IPv4 packet capture indexing and database system, built using open source technologies.

• A simple web GUI is provided for browsing,

searching, viewing and exporting PCAP data.

• Web APIs are accessible if you wish to design your

own GUI or directly grab PCAP with various

command line tools for further analysis or

processing.

• Find it on AOL’s GitHub page:

https://github.com/aol/moloch

It’s like AOL Search for PCAP repositories!

https://github.com/AOL/Moloch

WHAT IS MOLOCH NOT?

6

NOT IDS: NO ALERTS

NOT IPV6 (Today)

NOT SLOW

NOT CLOSED

NOT EXPENSIVE

WHY USE MOLOCH?

7

Real-time capture of network traffic for forensic and investigative purposes

• Combine the power of Moloch with other indicators (intelligence

feeds, alerting from IDS/anti-virus) to empower your analysts to

quickly and effectively review actions on the network to

determine the validity/threat.

• Review past network traffic for post compromise investigations.

Static PCAP repository

• Import large collections of PCAP that were created by malware.

• Import collections of PCAP from Capture The Flag events.

• Custom tagging of data at time of import.

THE PIECES OF MOLOCH

8

CAPTURE

• A C application that sniffs the network interface, parses the traffic, and creates the Session Profile Information (SPI data) and writes it to disk.

DATABASE

• Elasticsearch is used for storing and searching through the SPI data generated by the capture component.

VIEWER

• A web interface that allows for GUI and API access from remote hosts to browse or query SPI data and retrieve stored PCAP.

THE PIECES OF MOLOCH:CAPTURE

9

Libnids based daemon written in C

Can be used to sniff network interface for live capture

Can be called from CLI to do manual imports

Parses layers 3-7 to create SPI data

• Spits them out to the Elasticsearch cluster. A lot like making owl pellets!

THE PIECES OF MOLOCH:DATABASE

11

Elasticsearch (http://www.elasticsearch.org)

• Powered by Apache Lucene (http://lucene.apache.org)

• Requests over HTTP(s)

• Results returned in JSON

Nosql

• Network traffic doesn’t fit the mold for relational DBs.

Documented oriented

• Great for lots and lots of network sessions.

Automatic sharding across multiple hosts

• At the time, we skipped SOLR because it couldn’t run distributed.

Fast, scalable, all that goodness

THE PIECES OF MOLOCH:VIEWER

12

Node.js based application

• Event driven server side JavaScript platform.

• Based on Chrome’s JavaScript runtime.

• Comes with its own HTTP server and easy JSON for communication.

Web based GUI

• Browsing / searching / viewing / exporting SPI data and PCAP.

GUI and API use URIs

• All calls are done using URIs so integration with SEIMs, consoles, and command line tools is easy.

• Easy automation to retrieve PCAP or sessions of interest.


13

Nodejs based application

• Event driven server side JavaScript platform

• Based on Chrome’s JavaScript runtime

• Comes with its own HTTP server and easy JSON for communication

Web based GUI

• Browsing / searching / viewing / exporting SPI data and PCAP


• All calls are done using URIs so integration with SEIMs, consoles, command line tools is easy.

• Easy automation to retrieve pcap or sessions of interest.


14

Nodejs based application

• Event driven server side JavaScript platform

• Based on Chrome’s JavaScript runtime

• Comes with its own HTTP server and easy JSON for communication

Web based GUI

• Browsing / searching / viewing / exporting SPI data and PCAP


• All calls are done using URIs so integration with SEIMs, consoles, command line tools is easy.

• Easy automation to retrieve pcap or sessions of interest.

ARCHITECTURE OF MOLOCH:DATA FLOW

15

ARCHITECTURE OF MOLOCH:MULTINODE WITH CLUSTER

16

ARCHITECTURE OF MOLOCH:SCALE

17

Packets Captured Kilobytes Saved Sessions Saved

Example: Moloch Capture

Documents Disk Storage (MB)

Example: Elasticsearch

MOLOCH: SPI-DATATYPESSESSION PROFILE INFORMATION

18

IP

• Source

• Destination

• Ports

• Protocol

HTTP

• Method

• Status Codes

• Headers

• Content Type

DNS

• IP Address

• Hostnames

MOLOCH: SPI-DATATYPESSESSION PROFILE INFORMATION

19

SSL/TLS

• Cert Elements:

• Common Name

• Serial Number

• Alt Names

SSH

• Client Name

• Public Key

• Port

IRC

• Channel Name

• Hostname

MOLOCH: CAPTURECREATING SPI DATA

20


21


22


23


24

MOLOCH: DEMO

25

MOLOCH: QUESTIONS?

26

Technology

MOLOCH: Search for Full Packet Capture (OA Cyber Summit)