A Finite State Machine Model of TCP Connections in the Transport Layer

J. Treurniet and J.H. LefebvreDefence R&D Canada - Ottawa

Mike Hsiao, 20080822

Defence R&D Canada - OttawaTechnical MemorandumDRDC Ottawa TM 2003-139November 2003

2

Outline

Summary and Introduction Theory

– Transmission Control Protocol– TCP Finite State Machine Model– Detection of Anomalies

Implementation of the Algorithm Results and Discussion

– Network Management– Security– Research– False Positives

Conclusion and Future Work

3

Background – RFC & implementation

The protocols employed on computer networks are intended to be well-defined via Request for Comments (RFC) documents.

– In reality, there is enough freedom in the RFC specifications of the protocols that they can vary widely from one implementation to the next.

Additional information: http://lxr.linux.no/

Summary

4

Background – TCP

TCP is a reliable protocol, and its behavior follows a pattern that is somewhat predictable.

– The “flags” convey the commands that carry a connection through these stages.

…

SYNSYN-ACK

ACK

FINFIN-ACK

ACK

Three-way handshake

Data exchange

Closing

Diagram of a “textbook” TCP connection

Summary

5

Background – FSM

The authors model a TCP connection as an FSM,– defining the states of a FSM to reflect the stages of a

connection and– using the flags as the events that bring about transitions

among states.

The TCP FSM proposed by the authors

Summary

6

Background – Anomaly detection

To use the model for strict anomaly detection, a failure state is introduced to indicate the occurrence of a disallowed event or an attempted illegal transition.

– When the time-ordered flags of a TCP connection is input to the FSM,

if the connection enters a failure state or otherwise does not complete,

– the connection is flagged as anomalous.

State X

Failure State

Event Y

Summary

7

Background

The TCP FSM can be used to detect some network management issues and network security events, and it can also be used as a research tool to study the behavior of TCP on the Internet.

”Strict anomaly detection” defines a set of � � permitted events and detects activity which represents exceptions to those events.

– TCP, FSM, anomalous

Introduction

8

A snapshot of TCP anomalies

Impart information to– Network management

Unresponsive hosts Behaviors that lead to resource consumption

– Network security Scanning activity

The appeal of the model is in its simplicity and it utility in a wild range of applications

Introduction

9

Transmission Control Protocol

Theory

Note: Only certain flag combination are valid.

10

Transmission Control Protocol

Theory

Graceful Termination

In an ungraceful termination,the connection is torn down bytransmission of a reset (R)from either the client or the server.

On receiving an acceptable reset,if the receiving TCP is in 1) a non-synchronized state, => returns to LISTEN; 2) a synchronized states, => aborts the connection => informs its user.

11

In Practice

In practice, stages of a connection are only allowed to exist for a set amount of time, to free up resources that aren't being used.

in most Berkeley-derived systems

Theory

twice the max amount of time in RFC;some implement is one minute.

12

TCP client and server

The traditional state transition diagram for TCP is a graphical representation of two separate but related finite state machines, one for the client and one for the server.

The authors have simplified the traditional TCP state transition diagram to remove the differentiation of client and server.

Theory

13

Theory

Server

Client

Attempt toclose socket

Receive closerequest

Simultaneousclose

14

They proposed a 7-state TCP FSM.

15

6 events in the proposed TCP FSM

Theory

(e.g., SYN+FIN)

Such packets are either “crafted” or appear as a result of corruption of the packet.

16

Theory

almost 10% of all SYN packets are retransmitted

17

Detection of Anomalies

If the final state of the sequence is not Closed (C2), then the connection is flagged as anomalous.

For streams that terminate in an allowed state other than Closed, we denote the event as “timeout” to signify that the stream has �ended but the connection has not been closed.

Theory

18

Example of Anomalies

Denial of Service (DoS)– many H2 | timeout events for one host

– X | timeout or C1 | timeout

TCP half-open scan– H1 | RAPU, H1 | timeout, H2 | RAPU or H2 | timeout

Scans for OS fingerprinting (e.g., QueOS)

Theory

19

Traffic Traces

Unfiltered traffic is collected at the external interface of 3 class B networks using tcpdump.

Only the first 68 bytes of each packet were collected to obtain the full TCP header.

At peak times, an hourly file contained on the order of 2 million packets.

Implementation of the Algorithm

20

Implementation

The TCP FSM algorithm was written in MATLAB. In practice, each packet was read sequentially from

the raw traffic file.– Each socket pair was assigned a stream holding the

complete trace for that socket.– How to deal with the truncation of the traffic data? (i.e.,

streams that start or end in the middle of a connection)

August 21, 2000 data for 24 hours starting at midnight.

Implementation of the Algorithm

21

Analysis Results (1/3)

Results and Discussion

They do not proceed the normal synchronization process.

22

Analysis Results (2/3)

Results and Discussion

They do not complete the synchronization process.

23

Analysis Results (3/3)

Results and Discussion

24

Results and Discussion

1) Approximately 37% of the Internet TCP traffic does not obey the protocol.2) If we remove “SYN scan” and “bad flag scan”, ↓

25

Network Management

Unresponsive and non-existant hosts– H1 | RAPU, H1 | Timeout, H2 | RAPU and H2 | timeout

Delays (i.e., packets do not arrive in an appropriate time)

– L | {SA, APU, FAPU, RAPU}, H2 | RAPU and X | timeout Abandoned connections

– X | timeout and C1 | timeout Malfuncioning TCP

– H1 | FAPU

Backscatter effects– L | SA or L | SA

Results and Discussion

26

Security (1/2)

TCP connect and half-open scans– Open port: {S-SA-APU-RAPU} or {S-SA-APU-FAPU-APU-FAPU-APU}

– Half open: H2 | RAPU

– Closed port: H1 | RAPU, H1 | timeout

Stealth scans and inverse mapping (firewall)– SYN-FIN (e.g., Firewalls will be configured to block incoming

packets with only the SYN flag set)– L | APU + RST or ICMP destination unreachable– L | SA, L | FAPU, bad flag, L | RAPU

Results and Discussion

27

Security (2/2)

Aggregating scanning events– L | {SA, APU, FAPU, RAPU}, H1 | {RAPU, timeout} and H2 | {FAPU,

RAPU} or bad flag

Role of the Firewall– H2 | timeout– H2 | RAPU

Results and Discussion

1m 10m 1h 5h 12h 1d 2d 5d >5d

3

1

7

2

1 1

28

False Positives

By definition, all anomalies found using strict anomaly detection are anomalous and therefore there is no false positive rate to consider.– It is difficult to determine the intent of the user

who sends a SYN and receives a RST in response (failure H1 | RAPU).

– It has different meanings and possesses different importance to different applications.

Results and Discussion

29

Conclusions and Future Work (1/2)

Examining packet follows on a network that do not conform to the protocol

Successfully detect TCP-based scanning activity, delays, outages, misconfigurations and other unexpected TCP behaviour

Correlation engines could be used to correlate the events found through application of this model.

A successful full connect scan cannot be detected with this method because the flag pattern obeys the protocol specification.

Include an option for TCP traffic that is not officially allowed but is nevertheless present on the Internet.

IDS

Conclusions and Future Work

30

Conclusions and Future Work (2/2)

Using the proportionalities of anomalies found by implementing the TCP FSM, one can introduce failures using a Markov chain approach.

As well as a mechanism for traffic generation, Markov models may provide the capability of assigning a probability that a connection is valid.

This may be the basis of a statistical anomaly detection engine.

Conclusions and Future Work

31

Comments

The usage of a protocol can be considered as a distinctive signature to a program.

Our work also include service level, and an application is more complicated than TCP.

– More events (messages)– States are unclear.

Too much false positive for anomaly detection model.

Perfect attacks may exist.

32

Further Readings

TCP– Floyd, S. (2002). Inappropriate TCP Resets Considered

Harmful. (work in progress).– End-to-end Interest Maillist. Frequency of RST terminated

connections. USC Information Sciences Institute. Apps

– Sasha/Beetle (2000). A Strict Anomaly Detection Model for IDS. Phrack, 10(56).

– Das, K. (2002). Protocol Anomaly Detection for Network-based Intrusion Detection. SANS Institute.

– Lemonnier, E. (2001). Protocol Anomaly Detection in Network-based IDSs.