Flow Matrix Tutorial

8/3/2019 Flow Matrix Tutorial

1/23

FlowMatrix Tutorial

A message from the creators:

When developing FlowMatrix our main goal was to provide a better and more affordable network

anomaly detection and network behavior analysis tool for network security professionals. We chose to usea different technology for network intrusion detection than most of the modern IDS products currently use.

FlowMatrix uses signature free, self-tuning, multidimensional statistical network behavioral models to identify

network behavior anomalies, including unknown ones, in real time. In order to classify detected anomalies weuse combination of fuzzy logic, Bayesian networks and a few other techniques.

We tried to make FlowMatrix to stay focused on a specific security goal and keep it free of unnecessaryfunctionality. In our opinion many existing products attempt to cover too much ground diluting their main

purpose and burdening the user with an unnecessarily extensive configuration before their products can be

deployed.

Special considerations and efforts were given to achieve a low rate of false positives (FP) with high confidencelevel, the factors that plague many current IDS and put high workload on security analysts. The high rate of

false positives is a well known and hard problem to solve by any network anomaly tools, but we did our bestto achieve an acceptable balance between the detection rate and the false positive rate. In addition we are

constantly working to improve it.

Overall we believe weve mostly succeeded and we hope that you, our users, would find our product superior

and easy to use as well as more affordable than what is available on the market at this time. We are open for

suggestions from our users on how to improve the product to serve better its purpose. Please send yoursuggestions to [email protected]

FlowMatrix modus operandi

FlowMatrix consists of three main functional parts: the processing engine (backend), database and WebGUI

(front end).

The engine receives NetFlow records from capable sources, such as routers, switches, firewalls, etc. and

processes them in real-time for network and network applications anomaly detection. When installed the

engine first operates in learning mode. In this phase it is able to build network models with just a few hours ofnetwork data. It needs to be added that with so few initial data points available detected anomaly events can be

false positives. After the learning is complete (usually 7-14 days) it enters fully operational mode. In this mode

the computed network models are constantly compared with the current data derived from incoming

NetFlows. Alarms are generated if the models differ beyond allowed tolerances. The model and associatedthresholds are self-adjustable as theyre derived from the processed data itself (NetFlows) and not hardcoded

externally. Internal models are periodically updated without requiring dedicated periodic learning interval.

The models are built not just for whole network but also for 3 groups of user configurable applications and are

compared independently. This means that FlowMatrix can detect behavioral anomalies for applications which

can be defined by users. Applications are defined by groups of ports mostly used by applications in the group.


2/23

This will detect network applications anomalies even so network as a whole may operate correctly. Such

functionality is intended to monitor operation of some business critical network applications.

Detected network or application anomalies are classified when possible. Independently of classifications

detailed relevant information is presented such that manual classification can be performed

For the user, the model boundaries are expressed via thresholds, which if crossed, may trigger an anomalyevent generation. The engine is capable of detecting two major classes of anomalies: volume based and trafficcharacteristics based (number of unique source or destination ports, unique source or destinations IP addresses,

number of connections, connection properties etc.). Volume based anomalies are those caused by abnormal

number of IP packets, substantially lower or higher that usual. Traffic characteristics alarms are triggered by

unusual changes in distribution of one or more IP packet characteristics: src and/or dst addresses, src and/ordst ports.

Note: It is important to have network conditions close to typical when the product is first deployed to

compute the network and network application models closely matching the reality of your network. As result it

is advisable that you dont perform deployment of this product while you are under heavy attack, lite level ofattack is considered acceptable and will be compensated by the models. Having correct initial models

improves detection and lowers the False Positives rate.


3/23

Deployment Model

FlowMatrix is most effective when deployed properly, such that it receives NetFlow records about most

important traffic on your network. We recommend a deployment similar to that shown on the Figure 1.

Configure internal NetFlow sources that handle traffic from corporate hosts to Internet and vice versa such as

routers, switches and firewalls to export NetFlows to the FlowMatrix server. For best result and more visibilitymake sure those sources deal with clear, not NATed traffic. If you have multiple connections to Internet make

sure that traffic from all connections is observed and NetFlow records sent to FlowMatrix for processing.

Figure 1


4/23

Installation

Have the installation executable ready on the server where FlowMatrix is going to be installed

You can download latest version of FlowMatrix from this location:

http://www.akmalabs.com/downloads_flowmatrix.php

Launch the downloaded executable, in this example FMSetup.exe

Click Next on the Welcome

screen and select a destinationinstall folder, if appropriate


5/23

Click Install

The installer will check for existing Apache and PostgreSQL installations and will not proceed if any of those

is found on your system. FlowMatrix assumes that it owns installation of Apache and PosgreSQL servers

install will fail if it finds any of these servers installed. You would need to uninstall them and start theinstallation process over.

Otherwise, the install process will continue installing all necessary components and services

Note: All FlowMatrix components will be installed under the destination folder. No files are installed in thesystem directories

Once the install process successfully finishes press Ok to exit the installer.

To access the Web GUI click Start->All Programs->FlowMatrix->Flow Matrix. You can also create a shortcuton your desktop.

Note: FlowMatrix Web GUI is best when used with IE or Firefox browsers. Other browsers support may comein the future releases

.


6/23

Logon to the system by using default user name and password: admin / admin

Now you need to configure the product and NetFlow sources


7/23

Setup FlowMatrix and configure NetFlow sources

What is NetFlow?

NetFlow is a protocol used for collecting network traffic information, which was developed by

Cisco Systems, Inc.

NetFlow enabled devices, which include Cisco routers and switches (as well as switches and routers made by

other supporting vendors) generate records, which are sent from the router in UDP packets using NetFlow

protocol format. A NetFlow collector must then collect these packets as they stream from the router.

Some of the information that NetFlow provides per a reported flow are:

Source and destination IP addresses for the network flow;

Protocol type (field from IP header which indicates next protocol that follows);

Source and destination ports for the network flow when applicable for the protocol;

Total number of packets and bytes per flow;

Other information (TCP flags etc).

Note: Routers will only send the information pertaining to a given conversation after it has ended or when

timeout reached when it is configured so.

Examples of devices and vendors that support NetFlow

Cisco routers

Cisco switches (some models)

Juniper

Enterasys

Checkpoint (through 3rd

party modules)

In addition to commercial products listed above there are numerous implementation of NetFlow aggregators

that monitor network traffic in real time and produce NetFlow records as done by the routers and switches

from vendors above. Please check following link for good collection of free open source NetFlow probeswhich can be used as a NetFlow probe in case if you dont have NetFlow capable router/switch:

http://www.switch.ch/network/projects/completed/TF-NGN/floma/software.html
http://www.switch.ch/network/projects/completed/TF-NGN/floma/software.htmlhttp://www.switch.ch/network/projects/completed/TF-NGN/floma/software.html


8/23

Configure NetFlow receiving port on FlowMatrix

The default port FlowMatrix engine listens to NetFlow traffic is on port 2055. This can be changed in

Settings->System Options in the Web GUI

Example of configuring NetFlow on a Cisco router

Telnet to the router and enter the following commands in global config mode:

(config)#ip flow-export destination

(config)#ip flow-export version 5 peer-as

(config)#ip flow-export source

(config)#ip flow-cache timeout active 1

Note: Currently FlowMatrix only supports collection using version 5 of NetFlow.

Then for each interface you configured above enable NetFlow collection:


9/23

interface

ip route-cache flow


10/23

In short time (about 1-2 minutes) after NetFlow configured on both FlowMatrix and on the router, you can

verify that FlowMatrix engine receives the NetFlow traffic by looking on NetFlows recived (per minute lasthour) graph on the Dashboard->Summary View


11/23

Configure Learning Intervals (Optional)

Learning Intervals influence FlowMatrixs internal models and as a result user visible thresholds and affect

accuracy of anomaly detection. Typical default values provided should be appropriate for most deployments.

Current version defines four intervals which are day and night for weekdays and day and night for weekends.Unless your network activities differ substantially from default intervals you dont need to change them in the

initial setup and can change them later without braking the existing models which will be updated with newintervals in short time (1-2 hours) after your changes saved and models are reloaded at proper moments.

If your network usage intervals differ substantially and you would like to change defaults you should go to

Settings -> Learning Intervals page and adjust proper settings.

Click Save to commit changes


12/23

FlowMatrix Workflow

FlowMatrix interface was designed with a specific workflow in mind. Following this workflow will help you

to get most out of FlowMatrix capabilities. Later you most likely will workout your own workflow.

The recommended workflow starts at Dashboard->Summary View screen which provides an aggregate viewof various key events the system tracks.

To get familiar with main operational page, lets walk through the available graphs and explain their meaning.

- Anomaly Events Count graph shows total number of anomaly alarms per interval. In FlowMatrixdefault execution interval is 1 minute. This graph shows last 60 minutes of events. Please note that

graph shows only alarms and not warnings.

- Anomaly Events table shows summary of all anomaly alarms and warnings for last 60 minutes. You

can drill down on each of the alarms by clicking on summary description of the alarm. This will bring


13/23


14/23

We suggest that a security analyst would have the DashBoard summary view page in his view for continuous

monitoring. Things to pay attention in this mode are the Anomaly Events graph, Anomaly Events table and Packets Volume graph and the NetFlows received gauge. When the threshold is crossed there will be a

corresponding event in the Anomaly Events table. If the event is of some interest to you, click on it to go to the

Details page.

This page shows friendly summary of important flows for most anomalous virtual traffic clusters. The clustersfor which anomaly is very clearly represented will be classified and summary information about classificationwill be provided with most likely violators clearly stated. Please keep in mind that virtual traffic clusters do

not correspond to grouping of your hosts on network and determined based on IP addresses and number of

other parameters. If you would like to get more summary information you can drill it down by clicking at

links Show by IP count or Show by packet count.

Note: In two dimensional space an anomaly location is determined by its time (shown in the events table) and

the IP flow(s). Since at any given time multiple anomalies can occur or a single anomaly can spread multiple

IP flows the virtual clustering provides a way to separate affected IP flows from the unaffected ones


15/23

TheShow by IP count will show top src and dst hosts and ports sorted by a number of unique IP addresses

they communicated with. The Show by packet count will show top src and dst hosts and ports sorted bynumber of packets, i.e. volume.

Click on a link to see the cluster details screen:

There are five tables that help you to pinpoint the source of anomaly:

- Top 10 src IP addresses sorted by either IP or packet counts (depending on which link you clicked).

This table shows which unique dst IP addresses each src IP of the top 10 contacted to. For example, thescreen above shows that host 64.95.76.7 contacted 8 different hosts listed in the Dst IP Addr column.

For brevity, only 1 IP address is shown and all of them accessible when you click show list link.

Packet count and Kbytes count is also reflected in the corresponding columns.


16/23

- Top 10 dst IP addresses sorted by IP or packet counts. This table is the same as the Top 10 src IPs but

from the dst point of view. In our example, the first entries in both tables are reversed which indicatesan active pair of hosts talking to each other. However, depending on the attack (anomaly) it is not

always true that such a pairing would exists: when the traffic is asymmetric the dst or src host may not

even be in the top 10.

- Top 10 src ports and Top 10 dst ports have similar concept to that of Top 10 src IPs and Top 10 dst IPs

- Top 10 peers by packet count shows the 10 most verbose pairs of hosts exchanged the largest number

of packets. As was noted, in the case of asymmetric traffic information on a peer may not be available

in the Top 10 src and Top 10 dst tables so this table shows the peers regardless of the direction of the

traffic.

Typically, you would want to choose the option (by IP or Packet counts) that is relevant to the type of

anomaly. For example, if an anomaly is classified as SCAN type you would know that the Packet countoption is the optimal one as scans dont usually produce large number of packets. On the other hands, scans

would cover large number of IP addresses and ports so choosing by IP count option would give you more

information about the source or sources of the anomaly.


17/23

Once you have the tables in front of you additional information can be obtained regarding a particular IP

address or port. Click on an IP address to fetch whois type of details about the address. Clicking on a port willgive you IANA port assignment, if available for the port.


18/23

Aside from the Dashboard view there are two options to see all anomaly and warning events retained by the

system: Anomaly Events and Anomaly Analysis.

Anomaly Analysis allows you to see graphs for traffic characteristics and volume in two detection subspaces.

By default, last 12 hours are shown and updated automatically but you can select different time scale for the

live update if you wish. Click Go to start updating.To isolate an anomaly use the graphs time coordinate (X axis) and a time query.

For example, we want to see the first two traffic characteristics anomalies in the subspace #1 closely. We seethat the first occurred on Tuesday at roughly 1:43 (24-hours scale is used) and the second stopped at before3:06.


19/23

So, lets construct the query like this:

We can see the anomalies beginning and the end times as well as their nature (duration, magnitude, etc) more

precisely.


20/23

With the times noted we could go the Anomaly Events table to find the corresponding entries. Use the same

query to limit the number of events:

Our two events occurred at 1:44:35 and 2:17:35 (the third event is in subspace #2). Click on any of them to go

to the Event Details.


21/23

Other Tools of Interest

There a few auxiliary tools that facilitate anomaly detection and situation awareness. It is important to realize

that these tools can be used for standalone visual detection and also to verify and correlate system generated

anomalies.

Network Patterns->Traffic Characteristics Color Map provides color coded representation of various degree ofstandard deviation for Src/Dst IP addresses and ports grouped by virtual clusters. It covers last 300 minutes.The warmer the color the more the deviation of a particular characteristics in a cluster. The idea here is that

when a low intensity anomaly occurs (low intensity scan, etc) it should cause, a change in one of the four

traffic characteristics, i.e. deviation in one or more traffic clusters (row in the matrix). The deviation may be

too subtle for the system to detect it as an alarm. Very often this subtle anomaly manifests itself as having acertain pattern that can be easily spotted by the human eye. This feature requires some practice but can be an

invaluable auxiliary tool.


22/23

Network Patterns->Subspace #1 and Subspace #2 Harmonics graphs show raw harmonics of the network

computed from NetFlow. Usually, stable networks have some sort of periodicity in their harmonics thatchange if the network configuration itself changes. By visually determining if theres the periodicity has

changed you may spot otherwise undetectable changes. It is useful to look at these graphs using >1 day time

scale to assess any possible changes


23/23

Statistics->NetFlow Statistics assists in detecting anomalies that could be spotted by changes in the simple

stats such as Mean and Std. Deviation.

For example, in this picture we can see that a sudden spike in byte and packet counts may indicate a volume

based anomaly and it needs to be investigated. Again, use time scale and queries to spot or isolate anomalies.

Documents

Flow Matrix Tutorial