Tracing the Control Flow of Read/Write Operations in the Linux Kernel

Tracing the Control Flow of Read/Write Operations in

the Linux KernelMatt Weir

Our Original Goal

• To create a data logging system across the kernel with accurate timing that will monitor data as it moves up and down the data path.

The Feasibility of that Goal

The Current Goal

• Produce a framework that will assist in tracing the control flow of read/write operations in the Linux kernel using kernel markers

A Brief History of the Project

Week of May 26th3rd Week of Class

• Created our group• Decided upon our basic goals• Did research on previous efforts into this field

Question: Has this been done before?Answer: YesUnderstanding and Visualizing Full Systems with Data Flow Tomography

Week of June 2nd4th Week of Class

• Talked to Dr. Wang and various graduate students to try and figure out how file IO works in Linux

• This is a generalization from my own imperfect ability to fully follow the conversations but...– There’s a lot of mystery about how the current

version of Linux really works.• Started playing around with printk

First Experience using prinkJun 2 12:20:30 device85 kernel: DATATAGGING: Someone called kmalloc(message repeated 212341 times)

• Not so bad

Second Experience using prink

• Decided to add a timestampJun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at 123112321Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at 123128742Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at 123132342Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at 123132323Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at 123141424Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at 123164353Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at 123164353Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at 123173433Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at 123185454Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at 123198567Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at 123206566Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at 123209877Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at 123213421Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at 123223167Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at 123228744Jun 2 12:20:30 device85 kernel: DATATAGGING: "Someone called kmalloc at 123232148

Size of the Log File

In Defense of printk

• When I added them to multiple functions, it does show the control flow

• Can grep through the log file to get a smaller snapshot of what is going on

• No noticeable performance issues from the user standpoint

• They work

That being said…• It is hard to manage a large number of them• Adding/removing printks is time consuming• They require an external structure to turn

them “on/off” during run time– Didn’t even think of this option until I was using

markers• When inserting them make sure you don’t add

one right after an “if” statement that doesn’t use {}

Week of June 9th5th Week of Class

• Decided to move from printk to markers• Upgraded my kernel version to 2.6.26.6 so

that we would be using the same code• Dr. Baker walked through the control flow of

read statements with us• Figured out how to implement markers and

designed some basic test cases

Markers

• Added very recently to the Linux kernel• In the creator’s own words– “It makes sense to offer an instrumentation set of

the most relevant events occurring in the Linux kernel that can have the smallest performance cost possible when not active while not requiring a reboot of a production system to activate”

Adding Marker Support• In menuconfig– General->Activate markers

The Marker Structure

• The Marker– A hook in the code to call a function in an attached

probe• The Probe– A function that can be attached to markers

• The Manager– A kernel module that manages/arms and disarms

probes

Friday Night

• Started to worry since all we had was glorified prink’s

• Decided to have a few drinks…

Came up with an Idea

• Focus on the marker management kernel module

• Modify the marker code to support finer grained logging

• Try to trace the control flow in read/write statements

PirateAcorn• The management kernel module that I wrote

PirateAcorn (continued)

• Acorn– Counter Intelligence term: Slang for someone who

is performing traffic analysis• Pirate– Because they are way cooler than ninjas

PirateAcorn (continued)

• Manages all the probes via ioctl commands– Breaks up the probes into read and write groups– Can enable them individually or at the same time– Supports the ability to have additional groups

added to it– Can turn off monitoring for certain threads, such

as other logging programs– Can be set to monitor a specific thread or all

threads

Registering Probes

Arming the Probes

The Probe Function

Modification to marker.c

• Needed to add support so it would only fire if the marker was called by a thread that is being logged

• Didn’t want to put the check in the probe function since that was called only after the marker fires

• Instead made a quick function that checks to see if a marker should fire

The Marker Check Code

The Marker Code

Adding Markers• For the most part I concentrated on mapping

the VFS layer and the File system

Finding the current PID

• Most of the time it was easy– current->pid

• But in some cases I wasn’t allowed to reference current– seq_read() in linux/fs/seq_file.c– Called by vfs_read() as file->f_op->read

Possible Solutions• Just print out all calls to seq_read(), and filter

them out when processing the log file• Don’t bother to log seq_read() at all• Implement a binary value in marker.c that is set

true when a previous marker is allowed to fire, and false when a marker is denied

• Create a wrapper function• Include the PID value in a structure that is

already being passed to it

Well what values are passed to it?

• File->f_op->read(file, buf, count, pos)– Count and pos are integer values– Buf is the buffer from the user, really don’t want

to mess with that– What about file?

The File Structure

Yes it’s a Bad Idea

• But what would happen if I added a PID field?

Answer:PIRATEACORN: sys_read_start: Pid=3055 Time=500170187414PIRATEACORN: vfs_read_start: Pid=3055 Time=500170192998PIRATEACORN: vfs_read_fop_read: Pid=3055 Time=500170196783PIRATEACORN: seq_read_start: Pid=3055 Time=500170200492PIRATEACORN: sys_read_end: Pid=3055 Time=500170239089

Yes, though I can see possible issues with this implementation, the biggest being multiple threads accessing the same file

The Mystery Call

• This one still has me stumped• For several threads, such as metacity I hit a

roadblock when I trace their read control flow

The Mystery Call (continued)

• The current control flow goes– sys_read_start– vfs_read_start– vfs_read_fop_aioread– do_sync_read_start– do_sync_read_forloop– do_sync_read_end– sys_read_end

The Mystery Call (continued)

• I loose the trace in the following call in do_sync_read– Filp->f_op->aio_read(&kiocb, &iov, 1, kiocb.ki_pos)

• This means there is a aio_read function associated with an f_op that I don’t know about

It Should be Pretty Easy to Track that Down…

My Current Guess

• There is an non standard kernel module that is installed and has its own aio_read handler

• Metacity is the gnome window manger– I can see it doing some funky stuff

• When adding markers I found lots of similar examples where the control flow didn’t go as I thought it would

QUESTIONS / COMMENTS?