Standards Activities on Traffic Measurement

2

Outline

• Applications requiring traffic measurement

• Packet capturing and flow measurement

• Existing protocols

• IETF IPFIX and PSAMP working groups

3

Applications Requiring Traffic Measurement

(1)• Usage-based accounting

– input to charging and billing– various business model

• time-based, volume-based, QoS class-based• per application, per user, per user group

• Traffic engineering– optimizing network usage– traffic analysis on congested links

• origin of traffic• type of traffic• dynamic behavior (bursty, adaptive, …)

• Traffic profiling

4

Applications Requiring Traffic Measurement

(2)

• QoS monitoring– (passive) measurement of QoS properties– validating Service Level Agreements

• Attack detection and analysis– detecting (high volume) traffic patterns– investigation of origin of attacks

• Intrusion detection– detecting unexpected or illegal packets

• …

5

Outline

• Applications requiring traffic measurement

• Packet capturing and flow measurement

• Existing protocols

• IETF IPFIX and PSAMP working groups

6

The General Traffic Flow Measurement Process

Classification &Flow Recording Store

(TCPdump)Observation

Point

PAYLOAD HEAD PAYLOAD HEAD

PAYLOAD HEAD PAYLOAD HEAD

PacketCapturing

Filtering

Display(Ethereal)

Sampling

Visualize(FlowScan)

Analysis by applications

(TE, attack detect., QoS monitoring,

accounting, …)

… other …

packets

Filtering

Samplingflow records

flow records

packets

packets

flow recordsflow

records

7

The General Traffic FlowMeasurement Process

• Packet capturing at observation point• Packet sampling and filtering

– both steps may be trivial (1:1 sampling, no filtering) – both steps may be applied repeatedly

• Packet classification, mapping to flow records, maintaining of flow records

• Flow record sampling and filtering– both steps may be trivial (1:1 sampling, no filtering) – both steps may be applied repeatedly

• Processing flow records in application

8

Filtering

Samplingpackets

packets

Capturingpackets

Packet Capturing Protocols

• Packet Capturing Protocols– Capture packets at

observation point– Optionally: Sample and

filter packets– Export packets or parts of

packets (e.g., first 100 bytes)

– Packet classification, flow recording and processing after transfer

• Proprietary: sFlow• Standard (to be): PSAMP

packet transfer

Classification &Flow Recording

Application

flow records

rout

er

or p

robe

9

Filtering

Sampling

packets

Capturing

Flow Monitoring Protocols

• Flow Monitoring Protocols– Capture packets at

observation point– Optionally: Sample and

filter packets– Classify packets and

update flow records– Export flow records– Flow record processing

after transfer

• Proprietary: NetFlow, LFAP, CRANE

• Standard: Meter MIB, IPFIX flow record transfer

Classification &Flow Recording

Application

flow records

Filtering

Sampling

rout

er

or p

robe

10

Comparison

• Packet Capturing Protocolso simple function on router

or probe+ low cost on router or probe

– high data volume for packet transfer orunreliable recording because of sampling

– packet classification required after data transfer

• Flow Monitoring Protocolso more complex functions

on router or probe– high resource

requirement on router or probe: fast memory for flow records

+ low data volume for flow record transfer

+ flow records available after data transfer

11

Outline

• Applications requiring traffic measurement

• Packet capturing and flow measurement

• Existing protocols

• IETF IPFIX and PSAMP working groups

12

Protocols

• Packet Capturing– sFlow (InMon Corporation, HP spin-off)– PSAMP (under standardization at IETF)

• Flow Monitoring– LFAP (Riverstone)– CRANE (XACCT)– NetFlow (Cisco)– IPFIX (under standardization at IETF)– RTFM Meter MIB (IETF standard)– RMONMIB (IETF standard)

13

Data collector

sFlow

• By InMon Corporation• Includes packet capturing,

sampling and packet transmission• Statistical packet sampling• Timestamping at data collector• Configuration by sFlow MIB• RFC 3176, www.sflow.org• Applicable to high speed links

when sampling is used• Adopted by many vendors (HP, Hitachi,

Alaxaia - by Hitachi and NEC, Extreme and more)

sMon Meter

Application

Packets

14

Data collector

PSAMP

• Under standardization at IETF Packet Sampling WG

• Very similar to sFlow• Time stamping by meter• Configuration by PSAMP MIB• Intention to use IPFIX protocol

for packet transferPSAMP Meter

Application

Packets

15

LFAP• Light-weight Flow Accounting Protocol• Proprietary by Riverstone (Cabletron)• Just data transfer protocol• Meter at Connection Control Entity (CCE)

communicates to Flow Accounting Server (FAS)

• Tight and reliable interaction between CCE and FAS

• Reliable data transport• Flexible TLV coding of transferred data• Larger overhead than NetFlow • More cost-intensive at meter/CCE

and at data collector/FAS

CCE

Application

FAS

Flow records

16

CRANE

• Common Reliable Accounting for Network Element (CRANE) Protocol

• Proprietary by XACCT• Just data transfer protocol• Template-based data model• Focus on reliability• Not yet in extensive commercial use

17

Data collector

IPFIX

• Under standardization at IETF IP Flow Information eXport WG

• Very similar to NetFlow version 9• Will not use UDP, but use TCP or

SCTP (Stream Control Transmission Protocol)

• Standardization close to completion• Close collaboration with PSAMP WG

IPFIX Meter

Application

Flow records

18

Router

NetFlow• Proprietary by Cisco, but de-facto standard• Fast and efficient, implemented for IOS• Configurable measurement per 5-tuple• Unreliable data transport (UDP)• Hardware-supported on some models• Not well documented

– re-engineered by Juniper• Versions 1, 3, 5, 7, 8

– fixed data model– no support of IPv6 flows

• Version 9 (starting point for IPFIX standard)– data model templates– can report IPv6 flows– optional reliable transport– not related to older versions!– RFC 3954

Meter

Data collector

Application

Flow records

19

Real-Time Flow Measurement (RTFM)

• Very flexible and powerful meter– programmable rule sets– can serve several readers– programmable overload behavior

• Reader polls meter• Realization by SNMP Meter MIB• Free software implementation

NeTraMet• No acceptance at manufacturers• Complicated to use (too powerful)• Specified by RFCs 2720 - 2724

Meter

Manager

Reader

Application

Flow records

Config.

20

Remote Network Monitoring MIB (RMON)

• Very flexible and powerful• Serves more general goals (analysis on layers 2-4)

• Just a monitoring tool, no measurement architecture defined

• Suited for very specific analysis tasks• High (hardware) performance requirements• Too complicated and too expensive for massive

usage in routers• Specified by RFCs 2021(RMON2), 2613,

2819(RMON), 2895, 2896, 3144, 3287, 3273, 3395, 3434, 3577, 3729, 3737, 3919, 4149, 4150

21

Outline

• Applications requiring traffic

measurement

• Packet capturing and flow measurement

• Existing protocols

• IETF IPFIX and PSAMP working group

22

IETF IPFIX Working Group

• IP Flow Information eXport (IPFIX)– BoF sessions 12/00 and 08/01– active since 10/01

• Successor of RTFM (Real-Time Flow Measurement) WG

• Target (official): standardizing current practice– Target (unofficial): standardizing (something like) Cisco

NetFlow• Chairs

– Nevil Brownlee, CAIDA– David Plonka, University of Wisconsin

23

IPFIX Scope and General Requirements

Goal: Find or develop a basic common IP Traffic Flow measurement technology to be available on (almost) all future routers

• Fulfilling requirements of many applications• Low hardware/software costs• Simple and scalable• Metering to be integrated in general purpose IP

routers and other devices (probes, middleboxes)• Data processing to be integrated into various

applications• Interoperability by openness or standardization

24

Application

IPFIX Architecture

Flow Record

Observation Point

Flow Information Export

PAYLOAD HEAD PAYLOAD HEAD

PAYLOAD HEAD PAYLOAD HEAD

PAYLOAD HEAD PAYLOAD HEAD

PAYLOAD HEAD PAYLOAD HEAD

MeteringProcess

ExportingProcess

CollectingProcess

25

IPFIX Devices

OM

E

Probe

OM

E

Simple Router

O OO OM

E

Complex Router

O OO OMO OO O

M

E

Multiple Exporters

O OO OM

E

O OO

OM

E

Protocol Converter

(Meter MIB)

OM

E

OM

E

OM

E

M E

Concen-trator

C E

Proxy

C …M: MeterE: ExporterC: Collector

26

IPFIX WG: Expected Output

• Planned documents– Requirements RFC (completed, RFC 3917)– Evaluation RFC (completed, RFC 3955)– Protocol specification (in progress)– Data Model (in progress)– Architecture RFC (in progress)– Information model RFC (in-progress)– Applicability RFC (in-progress)

• No new protocol development in working group• Instead: protocol selection and refinement• Selected protocol: NetFlow version 9• Configuration of measurements

will not (yet?) be standardized

27

IPFIX WG: Current Status

• Good support from IESG (Internet Engineering Steering Group)

• High interest from equipment manufacturers– Cisco designed NetFlow v9 compliant to IPFIX

requirements and contributes to documents– Riverstone/Enterasys contributing actively– Juniper is closely monitoring progress

• Several accounting and billing system providers are monitoring and contributing– HP, XACCT, InMon, ...

• More information at http://ipfix.doit.wisc.edu

28

IETF PSAMP Working Group

• Packet SAMPling (PSAMP)– BoF session 03/02– active since 07/02

• Initiated by Nick Duffield, AT&T• Target: standardizing new technology for sampling,

filtering and exporting packets– can be interpreted as a component of the IPFIX

measurement process– but different to IPFIX, there is no current practice

• Chairs– Juergen Quittek, NEC

29

PSAMP Scope and General Requirements

Goal: Develop effective but low-cost packet sampling technology

• Allowing measurements at high-speed links• Fulfilling requirements of applications using per

packet measurement– QoS analysis, traffic profiling

• Very low hardware/software costs• Much simpler than IPFIX• Will use subset of IPFIX protocol• Metering to be integrated in general purpose IP

routers and other devices (probes, middleboxes)• Configuration of sampling included (different than

IPFIX)

30

Application

PSAMP Architecture

Packet Record

Observation Point

Packet Information Export

PAYLOAD HEAD PAYLOAD HEAD

PAYLOAD HEAD PAYLOAD HEAD

PAYLOAD HEAD PAYLOAD HEAD

PAYLOAD HEAD PAYLOAD HEAD

Sampling & Filtering Process

ExportingProcess

CollectingProcess

31

PSAMP WG: Expected Output

• Planned documents– Architecture RFC (in progress)– Packet Sampling and Filtering Spec. RFC (in progress)– Report Format and Protocol specification (close to final

document)– PSAMP MIB RFC (close to final document)– Applicability RFC (not started)

• Dependencies on IPFIX protocol development

32

PSAMP WG: Current Status

• Good support from IESG (Internet Engineering Steering Group)

• Growing interest from equipment manufacturers– Main drivers are AT&T, Cisco and NEC– Avaya is actively contributing– Alcatel, Avici, InMon, Lucent are monitoring and joining

discussions– Cisco shows strong interest in having PSAMP close to

IPFIX in order to re-use their existing IPFIX software