Network processing by pid

Energia Open Source

Network Monitoring

by process id

Nuno Martins

[email protected]

Energia Open Source

27.02.2012 Network Monitoring by Process id 2

Monitoring

Understand programs' actual behaviour when running

Analyse resource usage

Create a usage profile, to evaluate performance and correctness

Can be done actively (polling the monitor) or passively

Through dynamic or static instrumentation

Analysis of the data can be done online (during capture) or offline (post mortem)

Energia Open Source


Network Monitoring Network monitoring is done passively by capturing packets

Generally it is done using PCap (Packet Capture) Library with LSF (Linux Socket Filtering)

Analysis of communication protocols Analysis of the interactions between distributed entities Error detection, performance rating, troubleshooting, etc.

The reduction in obtained data is done through filters This way the overhead is reduced (because it doesn't have to copy so much data) Only capture the relevant packets for analysis

Energia Open Source


Actual Network Monitoring

NICHardware

Kernel

User

Network

TCPDump

LibPCAPUser APP User APP User APP

Before sending or receiving packets

TCP/IP stackPACKET stack

...

Energia Open Source


Actual Monitoring Mechanism

A filter is a set of rules These rules can be based on ports, addresses, protocols, etc. Or on a specific set of bytes Rules are combined with logical operands

Changing the filter with libpcap Necessary to drain the socket Possible to lose packets

When filters are more complex they can't be applied on kernel space All packets are captured and the filter is applied in userspace

Without specifying a filter the monitor will capture all packets

Energia Open Source


Challenges of network monitoring by process id

Monitoring the network activity of a process in user space is limited to some cases and can overload the system

Changing the filter by using libpcap has non-negligible latency

pcap filter Compile andoptimise

Draining and attach the new

filter

Latency when attaching a new filter0 Time

Energia Open Source


Network Monitoring by Process ID

New Approach towards the reduction of data for analysis More specific analysis

Performance and overhead issues When we only want a subset of the packets flowing on the network card

Can simplify the use of bpf filters

Energia Open Source


Why we need this

Analysis of closed protocols No access to source code Not easy to understand / follow some network protocols

Debug protocols when creating new ones

Determining if a program is leaking information

In production machines Cannot stop servers for debug Troubleshooting specific threads

Energia Open Source


Kernel Space Changes

Two Main Parts Main kernel code

Created a hook to be attached by the filter function Changed the filter_function at the end to call the hook

Module (MRoP – “Monitorização de Rede orientada ao Processo” [Pt])

KProbes handler functions Repository (a RB-Tree) Filter function User space interface (through DebugFS)

Energia Open Source


Kernel Components

The Kernel module developed has 4 components Syscalls hooked handlers

Repository (of socket information) ports and addresses

Filter function

DebugFS interface Communication to/from user land Statistics and control

Energia Open Source


New Filter Mechanism (by process id)

DebugFS

PID, stats, etc

Kernelfilter

module

Monitor Apptcpdump

Root userApp

User app

Kernel

User space

Packetstack

Sendingor receiving

packets

Energia Open Source


TCPDumpPcap LibraryMonitor

Control API

KProbes

Packet Filter Function

Repository

InstrumentedSyscalls handlers Hook

AF_PACKET

GenericProcess

NIC Driver

NIC

Packets

Energia Open Source


KProbes

Dynamic Instrumentation Mechanism on kernel space Different types of instrumentation based on what want to be

achieved Kprobe, Jprobe, KretProbe

int 3 instruction (trap) Does not need Debug information Uses kallsyms

To locate symbols Overhead of 0.6 microseconds

per probe hit It's a mechanism not a tool

Energia Open Source


Syscalls hooked handlers Connect, accept, bind, recvfrom, sendto and sock_close

function

KProbes is a kernel mechanism for instrumentationKRetProbes are probes that get the return value of the functionsKRetProbes use a trampolin to catch the return value

Inside the handlers the computation must be very quick

Energia Open Source


Syscalls hooked handlers (II)

Inside handlers Need to get the socket information The socket's information is on the parameters of the syscall or on socket descriptor The registers have the value of the socket descriptor The socket information is written to the repository

The are two handlers, one on the entry of the function and the other on the return This way on the return handler using the return value we are sure if the call was successful, if not the information on the repository regarding that socket is removed

Energia Open Source


Filter Function Packets that will be accepted by the bpf program filter are

evaluated by the module filter function The filter function only evaluates TCP and UDP packets Search the repository for the packet port

If it finds verifies the protocol and the address

Energia Open Source


Repository

This repository is for socket information so that the filter can know if a specific packet belongs to the target process

Implemented using a Red and Black Tree Mainly performance (must be searched once per packet)

Energia Open Source


Repository II

Energia Open Source


DebugFS interface to user land Created a directory on the DebugFS

Files for controlling the filtering mechanismSearch the process structure and add information to the repositoryClear the repositoryIdentify which process to monitor

Files for statistics/logging purposesHow many packets passed/dropped by the filterHow many times the handlers functions were calledHow many elements have the repository

Energia Open Source


Evaluation

Functional evaluation Created small and specific programs to verify socket information on both sides (user space and kernel space)

Transfered data using http, ftp and iperf protocols

Data transferred with http and ftp was monitored and saved to a file

It was visualized on Wireshark Application layer data was compared with the data sent using md5 and sha1 checksums

Energia Open Source


Evaluation II Performance Evaluation

Evaluated the overhead using a 1GB transfer While transferring data it was monitored and saved to a file

Overhead introduced: Dynamic instrumentation Managing the repository

Energia Open Source


Performance Evaluation Two machines connected directly on a 100 Mbit/s link Transferring 1GB data through that link using:

ftp, http protocols and iperf tool

Capturing only one flow of data

Capturing two flows of data (one being the relevant one) Capturing only one flow using MRoP

VS Capturing two flows

Measured times: Without monitoring (transfer only) Using standard monitoring Using MRoP (tcpdump + developed kernel module)

Energia Open Source


Performance Evaluation II

Energia Open Source


Performance Evaluation III

3.5%

Energia Open Source


Network Monitoring by Process ID

Created a new kernel module to extend functionality of network monitoring of a Process

Only captures the relevant packets for analysis

Low overhead

Maintains compatibility with old bpf filters

Shows better results when the analysis is focused on a subset of the network packets

Energia Open Source


More Integrated (Work in Progress)

This approach has some integration issues It is necessary to use the debugfs to introduce process ids Not integrated with libpcap Filters still don't have a pid mnemonic

The core functionality will be kept

Energia Open Source


TCPDumpPCap LibraryMonitor

Control API

KProbes


Repository


AF_PACKET

GenericProcess

NIC Driver

NIC

Packets

Energia Open Source


TCPDumpPcap LibraryMonitor

Control API

KProbes


Repository


AF_PACKET

GenericProcess

NIC Driver

NIC

Packets

(as debug)

Control API

Energia Open Source


Final Considerations

These changes benefit the new monitoring system based on process id

The mechanism of applying a new filter function may also benefit other kernel developers

Wireshark developers desire a mechanism to filter packets based on application id (they have it in their wish list)

Maybe it can be used to detect malware (since the instrumentation is done below the userspace)

Technology

Network processing by pid