Assessment of Data Path Implementations for Download and Streaming Pål Halvorsen 1,2, Tom Anders Dalseng 1 and Carsten Griwodz 1,2 1 Department of Informatics,

Assessment of Data Path Assessment of Data Path Implementations for Implementations for

Download and StreamingDownload and Streaming

Pål Halvorsen1,2, Tom Anders Dalseng1 and Carsten Griwodz1,2

1Department of Informatics, University of Oslo, Norway2Simula Research Laboratory, Norway

International conference on distributed multimedia systems (DMS’05), Banff, Canada, September 2005

2005 Pål Halvorsen, Tom Anders Dalseng & Carsten Griwodz

DMS’ 05, Banff, Canada. September 2005

Overview

Motivation

Existing mechanisms in Linux

Possible enhancements

Summary and Conclusions



Delivery Systems

Network

bus(es)



file systemcommunication

system

application

user space

kernel space

bus(es)

Delivery Systems



Pentium 4Processor

registers

cache(s)

I/Ocontroller

hub

memorycontroller

hub

RDRAM

RDRAM

RDRAM

RDRAM

PCI slots

PCI slots

PCI slots

network card

disk

file system

communication system

application


system

application

disk network card

Intel Hub Architecture several in-memory data movements and context switches



Motivation Data copy operations are expensive

consume CPU, memory, hub, bus and interface resources (proportional to data size)

profiling shows that ~40% of CPU time is consumed by copying data between user and kernel

gap between memory and CPU speeds increase different access times to different banks

System calls makes a lot of switches between user and kernel space




system

application

user space

kernel space

bus(es)

data_pointer data_pointer

Basic Idea of Zero–Copy Data Paths



Motivation Data copy operations are expensive

consume CPU, memory, hub, bus and interface resources (proportional to data size)

profiling shows that ~40% of CPU time is consumed by copying data between user and kernel

gap between memory and CPU speeds increase different access times to different banks

System calls makes a lot of switches between user and kernel space

A lot of research has been performed in this area!!!! BUT, what is the status today of commodity operating

systems?

Existing Linux Existing Linux Data PathsData Paths



Content Download


system

application

user space

kernel space

bus(es)



Content Download: read / send

application

kernel

page cache socket buffer

applicationbuffer

read send

copycopy

DMA transfer DMA transfer

2n copy operations 2n system calls



Content Download: mmap / send

application

kernel


mmap send

copy


n copy operations 1 + n system calls



Content Download: sendfile

application

kernel


sendfile

gather DMA transfer

append descriptor

DMA transfer

0 copy operations 1 system calls



Content Download: Results

UDP TCP

Tested transfer of 1 GB file on Linux 2.6 Both UDP (with enhancements) and TCP



Streaming


system

application

user space

kernel space

bus(es)



Streaming: mmap / send

application

kernel


application buffer

mmap uncork

copy


2n copy operations 1 + 4n system calls

copy

sendsendcork



Streaming: mmap / writev

application

kernel


application buffer

mmap writev

copy


2n copy operations 1 + n system calls

copy

Previous solution three less calls per packet



Streaming: sendfile

application

kernel


application buffer

DMA transfer

n copy operations 4n system calls

gather DMA transfer

append descriptor

copy

uncorksendfilesendcork



Streaming: Results Tested streaming of 1 GB file on Linux 2.6 RTP over UDP

TCP sendfile (content download)

Compared to not sending an RTP header over UDP, we get an increase of 29%(additional send call)

More copy operations and system calls required potential for improvements

Enhanced Streaming Enhanced Streaming

Data PathsData Paths



Enhanced Streaming: mmap / msend

application

kernel


application buffer

DMA transfer

n copy operations 1 + 4n system calls

gather DMA transfer

append descriptor

copy

msend allows to send data from anmmap’ed file without copy

mmap uncorksendsendcork msend

copy

DMA transfer

Previous solution one more copy per packet



Enhanced Streaming: mmap / rtpmsend

application

kernel


application buffer

DMA transfer

n copy operations 1 + n system calls

gather DMA transfer

append descriptor

copy

mmap uncorksendsendcork rtpmsend

RTP header copy integrated intomsend system call

previous solution require three more calls per packet



Enhanced Streaming: mmap / krtpmsend

application

kernel


application buffer

DMA transfer

0 copy operations 1 system call

gather DMA transfer

append descriptor

copy

krtpmsend

previous solution require one more call per packet

An RTP engine in the kernel adds RTP headers

rtpmsend

RTP engine

previous solution require one more copy per packet



Enhanced Streaming: rtpsendfile

application

kernel


application buffer

DMA transfer

n copy operations n system calls

gather DMA transfer

append descriptor

copy

rtpsendfile

existing solution require three more calls per packet

uncorksendfilesendcork

RTP header copy integrated intosendfile system call



Enhanced Streaming: krtpsendfile

application

kernel


application buffer

DMA transfer

0 copy operations 1 system call

gather DMA transfer

append descriptor

copy

krtpsendfile

previous solution require one more call per packet

An RTP engine in the kerneladds RTP headers

rtpsendfile

RTP engine

previous solution require one more copy per packet



Enhanced Streaming: Results

Tested streaming of 1 GB file on Linux 2.6 RTP over UDP

TCP

send

file

(con

tent

dow

nlo

ad)Ex

isting

mec

hani

sm

(str

eam

ing)

mmap based mechanisms sendfile based mechanisms

~27%

impr

ovem

ent

~25%

impr

ovem

ent



Conclusions Current commodity operating systems still pay a high

price for streaming services

However, small changes in the system call layer might be sufficient to remove most of the overhead

Conclusively, commodity operating systems still have potential for improvement with respect to streaming support

What can we hope to be supported?

Road ahead: optimize the code, make patch and submit to kernel.org



Questions??

Documents

Assessment of Data Path Implementations for Download and Streaming Pål Halvorsen 1,2, Tom Anders Dalseng 1 and Carsten Griwodz 1,2 1 Department of Informatics,