Using Model-Checking to Debug Device Firmware

Using Model-Checking to Debug Device Firmware

Sanjeev KumarMicroprocessor Research Labs, Intel

Kai LiPrinceton University

Using Model Checking to Debug Device Firmware 2

Programmable Devices

Network Card

Bus

CPU

D

MemDisk

Network

CPU

D

D

D

Mem

Main CPUMain CPU Main MemoryMain Memory

Move functionality from main CPUs to devices

Device firmware is more complex


Firmware for Programmable Devices

• Difficult to write and debug– Use concurrency

• Inherently difficult to program correctly

– Optimized for high performance• Tradeoff program simplicity for performance

– Limited debugging support available

• Firmware reliability is important– Firmware is trusted by the OS– Bugs in firmware can crash the entire machine

• Model checking is a promising approach


Model CheckingSystematically verify properties of concurrent systems• Using State-space exploration

– Try all possible scheduling options

• Advantages– Automatic – Produces counter example

• Disadvantages– Computationally expensive

• Exponential search

• Models cannot be too big

Model Test Code

Program


Using Model Checking to Debug Firmware

• Extracting models from programs– Manually (by the programmer)– Automatic (using the compiler)

• Reduces programmer effort• Reduces mismatch between program and model

• Extracting smaller models models– Support for abstraction

• Discard irrelevant details in the program


Our Work

• Extract abstract models using a compiler– General compiler techniques

• Programmer controls the abstraction process– Specifies what needs to be abstracted

• Compiler performs the abstractions conservatively

– Extract models for Spin model checker from programs written in the ESP language

– Make practical choices• Goal: Debugging and not Verification

• Used to debug VMMC firmware for a network card– Found 7 bugs that can cause the firmware to deadlock

• Could not find these bugs without support for abstraction


Related Work

• Manual Model Extraction– Harmony, RUBIS, Plan 9, Fluke OS

• Used model checking to debug a subsystem

• Automatic Model Extraction– Teapot, Promela++, Esterel, Java Pathfinder

• Domain-specific and general-purpose languages

• Automatic Extraction + Support for abstraction– Feaver, Lie et. al., Bandera


Outline

• Background• Extracting Abstract Models using a Compiler

– From ESP Language– For Spin Model Checker

• Evaluation: Debugging VMMC Firmware• Conclusion and Future Work


ESP: A Language for Programmable Devices

pgm1.spin

pgmN.spin

pgm.C

ESP CompilerESP Compilerpgm.ESP

help.CGenerate

Firmware

Develop

and

Test

using

Model

Checker

test1.spin

testN.spin

Goals1) Easy to program2) Allow extensive testing3) Performance


The ESP Language

• Concurrent language: Processes & Channels– Pure message-passing communication– in, out, alt operations on channels– Channels are synchronous or unbuffered– Processes and channels are static

• A number of interesting features– Explicit memory management scheme that uses

model-checking to ensure safety– Supports dispatch on channels– Efficient and powerful interface to C


Extracting Models for Spin

• Step 1 : Detailed models [ PLDI’01 ]– Translate each language construct into Spin– Simple translation

• int, bool, records, arrays, unions• If-then-else, while-loops• process, channel

– Spin does not support• Dynamic memory allocation & Pointers• Additional bookkeeping necessary to support these

• Can be used to check local properties– Debug subsystems (1-2 processes) separately– Too big to debug the entire system


Extracting Abstract Models Conservatively

• Step 2: Abstract models– Necessary to check global properties (like deadlocks)– Drop unnecessary details

• Depending on the property being verified

– Programmer controls the abstraction• Abstraction specified by the programmer

– Drop variables– Drop fields from records and unions

• Compiler used the abstraction specified conservatively– Could introduce fast-positive bugs– All bugs in the programs will be present in the extracted model– Involves dealing with a number of tricky cases


Examples

$b2: boolean = true;

...

$b1: boolean = b2;

$b2: boolean = true;

...

$b1: boolean = b2;

type recT = #record of { int count; }type recT = #record of { int count; }

$r1: recT = {0};

if (b) { r2 = r1; }

...

r1.count = 5;

$r1: recT = {0};

if (b) { r2 = r1; }

...

r1.count = 5;

if

:: b1 = true

:: b1 = false

fi

if

:: b1 = true

:: b1 = false

fi

if

:: r2.count = 5

:: skip

fi

if

:: r2.count = 5

:: skip

fi

X

X

X

XX

Conservative: Use nondeterminism to broaden the state-space searched


Outline

• Background• Extracting Abstract Models using a Compiler• Evaluation: Debugging VMMC Firmware• Conclusion and Future Work


VMMC

• High-performance communication– Bypass OS for data transfers

• Used Myrinet network cards– Gigabit network– 33 MHz CPU, 1 MB memory

• Original VMMC firmware– Implemented in C

• Several man-years of debugging

• Still encounter bugs

• Some involve complex race conditions that are triggered only occasionally

Data

OS

Network Card

Application

Network


Debugging VMMC Firmware

• Reimplemented VMMC firmware using ESP• Used model checking to debug

– Global property of program (deadlocks)– Hard-to-find bugs– Found 7 bugs using abstract models

• 4 Bugs would cause deadlock during normal operations• 3 Bugs would be triggered only by a malicious machine

– Could not find these bugs without abstractions

• No firmware bugs encountered on device– Microbenchmarks– SPLASH2 parallel application suite

• On a 16-processor SMP cluster


Resource used for Model Checking

Spin ModeStates Searched

(in Millions)CPU Time

(in seconds)Memory

(in Mbytes)

Exhaustive 0.38 84.0 268.35*

Partial mode 99.7 14250.0* 167.92

• Only partial search was possible• Even partial searches were effective

* Limiting Resource

VMMC Firmware


Model extracted from VMMC Firmware

File Lines of Code

ESP Program 453

Abstraction Specification 108

Abstract Model Extracted 2202

Test Code 128

• Programmer only write a small amount of Spin Code• Program can be rechecked with little effort


Outline

• Background• Extracting Abstract Models using a Compiler• Evaluation: Debugging VMMC Firmware• Conclusion and Future Work


Conclusions

• Use compiler to extract abstract models– Evaluation: Debugged VMMC firmware

• Using compiler to extract models is good– Significantly reduces effort required to model check

• Abstraction is required– To check global properties like deadlocks

• Programmer can control the abstraction– Compiler is conservative– Does not require the programmer to be correct

• Only partial search was possible– Still effective in finding bugs


Future Work

• Optimizations to reduce size of state space– Eliminating more redundancies

• Quantify the effectiveness of a partial seach– Estimate the fraction of state-space searched

• Use type systems to reduce the size of state space that has to be searched


To find out more, Visit http://www.cs.princeton.edu/~skumar

Questions?


Debug not Verify

• Several sources of incompleteness and unsoundness remain– Programmer supplied Spin code– Partial model checking

• The goal is to isolate/reduce the unsound portions of the code


ABCDEFABCDEF

ABCDEFABCDEF

ABCDEFABCDEF

Abcdef Ghijk

Abcdef Ghijk

Abcdef Ghijk

Abcdef Ghijk

Abcdef Ghijk

Abcdef Ghijk

Documents

Using Model-Checking to Debug Device Firmware