On Application of Anomaly Detection in Network Securityusers.jyu.fi/~timoh/TIES327/anomaly2016.pdf · On Application of Anomaly Detection in Network Security Mikhail Zolotukhin Department

IntroductionTheoretical background

Example: analysis of HTTP logsAssignment: anomaly-based intrusion detection

Conclusion

On Application of Anomaly Detection in NetworkSecurity

Mikhail ZolotukhinDepartment of Mathematical Information Technology,

University of Jyvaskyla, Finland

30/11/2016

Mikhail Zolotukhin On Application of Anomaly Detection in Network Security



Conclusion

Table of contents

1 Introduction

2 Theoretical background

3 Example: analysis of HTTP logs

4 Assignment: anomaly-based intrusion detection

5 Conclusion




Conclusion

Introduction




Conclusion

Research motivation

Recent growth in the use of computer technologies has led tothe development of new means to automatically gather hugevolumes of diverse data

The analysis of the collected data is supposed to help insolving problems related to behavior and interactions ofhumans and machines

The main aim of this analysis process is to extract informationfrom a dataset and sometimes find patterns in data that donot conform to expected behavior




Conclusion

What is anomaly?




Conclusion

What is anomaly?

Anomaly - an item, event or observation which does notconform to an expected pattern or other items in a datasetAnomaly detection is applicable in a variety of domains,such as intrusion detection, fraud detection, fault detection,system health monitoring, event detection in sensor networks,and detecting Eco-system disturbances




Conclusion

Signature-based cyber attack detection

An attack against a computer or network system is detectedbased on a signature of the attack

The signature can contain a sequence of bytes or operationinstructions

This approach is accurate which makes it successful incommercial intrusion detection

This approach can not detect attacks for which it has notbeen programmed, and, therefore, can not detect zero-dayattacks




Conclusion

Anomaly detection vs Signature-based detection

Intrusion detection system learns behavior of normal users

Behavior patterns which deviate significantly from establishednorms are considered as anomalies

The system is modeled according to normal behavior and,therefore, is able to detect zero-day attacks

The number of false alerts will probably be increased becausenot all anomalies are intrusions




Conclusion

Example: SQL injection




Conclusion





Conclusion





Conclusion





Conclusion


Signature-based detection:

Vulnerability discovered: username=1’) UNION ALLSELECT 95,95,CONCAT(0x71626a7071,0x4c496778754c43...

Signature: username=1’) UNION ALL SELECT95,95,CONCAT(0x71626a7071,0x4c496778754c43...

Injection variation: username=2’) UNION ALL SELECT95,95,CONCAT(0x71626a7071,0x4c496778754c43...

Encoded injection: username%3d1%27%29%20UNION%20ALL%20SELECT%2095%2c95%2cCONCAT%280x71626a7071%2c0x4c496778754c43...




Conclusion


Anomaly-based detection:

username=alice in wonderland (normal)

username=juha in university (normal)

username=aleksi in pub (normal)

username=katja on stadium (normal)

username=marko in somewhere else (normal)

username=1’) UNION ALL SELECT 95,95,CONCAT(0x71626a7071,0x4c496778754c437 (anomaly, SQL injection)

username=mikko likes long usernames 123))))))1,2,3,4,5(anomaly, but not injection, i.e. false positive)




Conclusion

Theoretical background




Conclusion

Anomaly detection process

Anomaly detection is the computational process of discoveringanomalous patterns in large data sets involving methods at theintersection of artificial intelligence, machine learning, statistics,and database systems.




Conclusion

Preprocessing

The main aim of the preprocessing is to remove irrelevant andredundant information present in the data in order to extractknowledge from the data more accurately and reduce processingtime during later phases of the data mining process.

Feature extraction and selection

Standardization and normalization

Dimensionality reduction

The resulting product of data preprocessing is a training set fromwhich knowledge can be discovered.




Conclusion

Feature extraction

Size: length, height

Color: from (0,0,0) to (255,255,255) in RGB format

Direction: from left to right, from right to left




Conclusion

Feature selection

We can notice that all fishes have similar length/height proportion,which means that one of these features is not necessary:

Size: length, height

Color: from (0,0,0) to (255,255,255) in RGB format

Direction: from left to right, from right to left




Conclusion

Feature standardization

Simple example:

Normal data: x1 = [1, 0, 200], x2 = [2, 0, 250]

Center c = [1.5, 0, 225], radius r ≈ 25

Suspicious data y = [25, 0, 225]

Let’s check: distance(y,c) = 23.5 < 25

Is y normal?




Conclusion


Values of features might have different scales, the featurevectors are supposed to be standardized

For example, max-min normalization performs a linearalteration on the original data so that the values arenormalized within the given range, e.g. [0,1]

To map a value yij of an attribute (y1j , y2j , . . . , ynj) fromrange [min1≤i≤n yij ,max1≤i≤n yij ] to range [0, 1], thecomputation is carried out as follows

zij =

yij − min1≤i≤n

yij

max1≤i≤n

yij − min1≤i≤n

yij, (1)

where zij is the new value of yij in the required range.




Conclusion


Back to our simple example...

Normal data: x1 = [1, 0, 200], x2 = [2, 0, 250]

Normal data standardized: x1 = [0, 0, 0], x2 = [1, 0, 1]

Center c = [0.5, 0, 0.5], radius r ≈ 0.7

Suspicious data y = [25, 0, 225]

Suspicious data standardized (with regards to x1 and x2)y = [24, 0, 0.5]

Let’s check again: distance(y,c) = 23.5 > 0.7

y is an anomaly!




Conclusion

Dimensionality reduction

Data (3-dimensional): x1 = [1, 0, 200], x2 = [2, 0, 250],y = [10, 0, 225]

Value of the second feature is always zero.

The second feature is not necessary since it does not give usany valuable information

Data in new (2-dimensional) space: x1 = [1, 200],x2 = [2, 250], y = [10, 225]

Dimensionality reduction can be used to produce a compactlow-dimensional encoding of a given high-dimensional dataset orsimplify, reduce, and clean the data for subsequent analysis




Conclusion

Analysis

During the analysis step, we build a model which accuratelydescribes the structure of the data

This model is supposed to describe normal behavior patterns

The model can include length of values, pairwise distancebetween feature vectors, distance to feature vectors from theirmean value, reconstruction error after applying stackedauto-encoder, etc

Any pattern that deviates from defined norms is classified asanomalous

Types of anomaly: point, contextual, collective




Conclusion

Point anomaly

Point anomaly - an individual data object that is distinct withrespect to the rest of data set




Conclusion

Point anomaly

Point anomaly - an individual data object that is distinct withrespect to the rest of data set




Conclusion

Contextual anomaly

Contextual anomaly - data objects that are anomalous in a specificcontext but not in other situations




Conclusion

Contextual anomaly





Conclusion

Contextual anomaly





Conclusion

Collective anomaly

Collective anomaly - a collection of related data instancesanomalous with respect to the entire data set




Conclusion

Collective anomaly

Collective anomaly - a collection of related data instancesanomalous with respect to the entire data set




Conclusion

Algorithm validation

Checking that all anomalies in the dataset are foundaccurately with low number of false alarms (Accuracy)

Verifying that the normal user behavior model produced worksfor a similar dataset on which the algorithm was not trained(Reliability)

Analyzing the algorithm to determine its computationalresource usage (Efficiency)

Comparing the algorithm with existing analogues in terms ofaccuracy, true positive rate, false positive rate, detectionspeed, etc (Competitiveness?)




Conclusion

Example: analysis of HTTP logs




Conclusion

Analysis of HTTP logs

Computer networks and systems are vulnerable to differentforms of intrusions

One of the most popular attack targets is web-servers andweb-based applications

Usually, users of such applications request and sendinformation using queries, which in HTTP traffic are stringscontaining a set of attributes




Conclusion


08:51:19 190.165.160.29 GET /wp-content/style.css

08:52:45 85.161.110.55 GET /feed=comments-rss2

08:53:34 103.168.115.27 GET /page=55

08:55:14 85.119.123.145 POST /login.php?name=admin%27%29+−−+&password=+

08:55:57 192.168.160.29 GET /page=34

08:56:14 91.167.111.33 GET /uploads/lapland.png

14:00:05 10.66.14.17 POST /login.php?name=labra01



08:55:14 - SQL injection by 85.119.123.145,14:00:05 - Password bruteforce by 10.66.14.17.




Conclusion


HTTP logs

Data patterns Intrusion Detection System

new HTTP query Web serverLabel




Conclusion

Feature extraction by n-gram

A n-gram is a sub-sequence of n overlapping items (characters,letters, words, etc) from a given sequence.

Example

/resource?parameter1=value1&parameter2=value2.

/r, re, es, so, ou, ur, . . . , lu, ue, e2.

[47, 114], [114, 101], [101, 115], [115, 111], [111, 117],[117, 114], . . . , [108, 117], [117, 101], [101, 50].

2562 numeric vector. For example, the (256 × 61 + 118)-thentry in this vector contains the value equal to 2 since the pair[61, 118], which corresponds to pair ”=v”, can be seen twice.




Conclusion

Self-Organizing Map

The SOM is an unsupervised, competitive learning algorithm thatreduces the dimensions of data by mapping these data onto a setof units set up in much lower dimensional space.

SOM contains a regular grid of neurones each of which is fullyconnected to the input layer.

Each neuron of the SOM has an associated with ithighdimensional prototype.

At each training step, a sample vector from data set ismapped to the best matching prototype (BMU) of the SOM.

Prototype vectors are updated so that the BMU and itstopological neighbors are moved closer to the input vector inthe input space.




Conclusion

Self-Organizing Map

0 10

1

0 10

1

0 10

1

0 10

1

0 10

1

0 10

1




Conclusion

SOM limitations

1 The size and dimensionality of the SOM model is fixed priorto the training process and there is no systematic method foridentifying an optimal configuration.

2 SOM can not represent hierarchical relation that might bepresent in the data.

These limitations can be resolved by applying Growing HierarchicalSelf-Organizing Maps.




Conclusion

Growing Hierarchical Self-Organizing Map

GHSOM is a multi-layered hierarchical architecture which adaptsits structure according to the input data.

GHSOM is initialized with one SOM.

The first SOM grows in size until it achieves an improvementin the quality of representing data.

Each node in this map can dynamically be expanded down thehierarchy by adding a new map at a lower layer providing afurther detailed representation of data.

The procedure of growth can be repeated in new maps.




Conclusion

Detection of anomalous queries by GHSOM

Model TPR FPR Accuracy Precision

1-gram 89.0 % 0.01 % 99.4 % 99.5 %

2-gram 99.9 % 0.01 % 99.9 % 99.9 %

3-gram 100 % 0.01 % 99.9 % 99.9 %




Conclusion

Assignment: anomaly-based intrusion detection




Conclusion

Assignment: part 1

We analyze a small pcap-file which contains network trafficbetween a web service and its clients during two hours

The traffic is mostly legitimate but contains three attacks:two password brute-force attempts and one Sqlmap scan

Features selected: source IP address, source port, protocoland packet length

The detection is carried out based on calculation of sampleentropy




Conclusion

Sample entropy

Sample entropy allows one to capture the degree of dispersal orconcentration of the parameter’s distribution. Let us assume thatin the t-th time interval the i-th parameter has nti unique valueswhich appear with frequencies pti1, . . . , p

tinti

. In this case, sample

entropy E ti for the i-th parameter in the t-th time interval is

defined as follows:

E ti = −

nti∑k=1

ptik log2 ptik .

Entropy is equal to zero when all values are the same, and it takeson its maximal value when all values are different.




Conclusion

Anomalously high entropy

Entropy values below: 1) source IP address, 2) source port, 3)protocol, 4) packet length




Conclusion

Assignment: part 1

1 Using the figure obtained, explain why entropy values ofsource IP address, source port, protocol and packet lengthincrease during all three attacks detected (1p.)?

2 What causes false alarms recorded? Explain at least two ofthem (1p.).




Conclusion

Assignment: part 2

We analyze two small pcap-files which contains network trafficbetween a web service and its clients during few minutes

The first file contains only normal (legitimate) traffic and it isused as the training set

The second file contains normal traffic mixed with the trafficgenerated during Slowloris and it is used as the testing set

The detection is carried out based on k-nearest neighborsalgorithm with k = 1




Conclusion

Normal HTTP request

Request URL:http://jyu.fi/Request Method:GETAccept:text/html,application/xhtml+xml,application/xmlAccept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3Accept-Encoding:gzip,deflate,sdchAccept-Language:en-US,en;q=0.8Connection:keep-aliveUser-Agent:Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.22(KHTML, like Gecko) Ubuntu Chromium/25.0.1364.160Chrome/25.0.1364.160 Safari/537.22. . .




Conclusion

Slowloris

Attacker tries to initiate lots of connections with server

He tries to hold them open as long as possible by periodicallysending subsequent HTTP headers, adding to-but nevercompleting-the requests

Web server keeps these connections open, filling its maximumconcurrent connection pool, eventually denying additionalconnection attempts from clients




Conclusion

Slowloris request

Request URL:http://jyu.fi/Request Method:GETAccept:text/html,application/xhtml+xml,application/xmlafter some timeAccept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3after some timeAccept-Encoding:gzip,deflate,sdchafter some timeAccept-Language:en-US,en;q=0.8after some timeConnection:keep-aliveafter some time. . .




Conclusion

Feature selection

With the help of Wireshark, for each conversation between a clientand the server, we extract:

1 Total number of packets

2 Total number of bytes

3 Number of packets sent from the client to the server

4 Number of bytes sent from the client to the server

5 Number of packets sent from the server to the client

6 Number of bytes sent from the server to the client

7 Duration of the conversation




Conclusion

Normal and Slowloris connections

Blue dots - normal connections, red points - Slowloris connectionsAxis: total number of packets vs total number of bytes




Conclusion

Assignment: part 2

1 In Octave script used for Slowloris detection, select twodifferent features from the list by modifying value of variable”select features” (line 18) and re-run the script. Try at leastfive different combinations. Add to the report featurecombinations you selected and the resulting detectionaccuracy (1p.).

2 What combination of features from those five you tested inthe previous assignment gives the maximal detection accuracy(96.9 %)? Based on definition of Slowloris attack, try toexplain why this combination allowed you to get the bestresult (1p.).




Conclusion

Feature selection

For each conversation, we extract:

1 Duration of the conversation,

2 Number of packets sent,

3 Average, minimal and maximal size of packets,

4 Average, minimal and maximal size of TCP window,

5 Average, minimal and maximal time since the previous packet,

6 Average, minimal and maximal time to live,

7 Percentage of packets that have TCP flag SYN,

8 Percentage of packets that have TCP flag ACK,

9 Percentage of packets that have TCP flag PSH,

10 Percentage of packets that have TCP flag RST,

11 Percentage of packets that have TCP flag FIN.




Conclusion

Clustering

Clustering is a division of data into groups of objects withoutknowing the structure of the dataset

Each such group (cluster) consists of objects that are in someway similar between themselves and dissimilar to objects ofother groups

Methods: hierarchical clustering, centroid-based clustering,density-based clustering.




Conclusion

DBSCAN

DBSCAN is a powerful density-based clustering algorithm

DBSCAN starts with an arbitrary point that has not beenvisited. This points ε-neighborhood is found, and if it containssufficiently many points (more than Nmin), a cluster is started.Otherwise, the point is labeled as noise, although this pointmight later be discovered as a part of another pointε-environment and hence be made a part of a cluster

Then, a new unvisited point is processed, leading to adiscovery of a further cluster or noise




Conclusion

DBSCAN

00




Conclusion

DBSCAN

00




Conclusion

DBSCAN

00




Conclusion

DBSCAN

00




Conclusion

DBSCAN

00




Conclusion

Results

Table: Intrusion detection accuracy of different detection methods

Algorithm TPR FPR Accuracy

K-means 100 % 0.4878 % 99.9951 %

KNN 100 % 0.2091 % 99.9979 %

SVDD 100 % 6.0627 % 99.9390 %

SOM 100 % 0.4878 % 99.9951 %

DBSCAN 100 % 0.0697 % 99.9993 %




Conclusion

Conclusion




Conclusion

Conclusion

We studied what are anomaly and anomaly detection

Got familiar with the process of anomaly detection

Learned some anomaly detection methods

Considered several examples of finding network attacks basedon anomaly detection approach




Conclusion

Thank you

Thank you for your attention.

Questions?

Have a nice day!


Documents

On Application of Anomaly Detection in Network Securityusers.jyu.fi/~timoh/TIES327/anomaly2016.pdf · On Application of Anomaly Detection in Network Security Mikhail Zolotukhin Department