Configuration Coverage in the Analysis ofLarge-Scale System Software
Reinhard Tartler, Daniel Lohmann,Christian Dietrich, Christoph Egger, Julio Sincero
System Software Group
Friedrich-Alexander UniversityErlangen-Nuremberg
October 23, 2011
supported by
Configuration Complexity
Linux has become incredibly configurable
Complexity increases considerably
; Source of bugs!
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)2 – 10
Configuration Complexity
Linux has become incredibly configurable
Complexity increases considerably
; Source of bugs!
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)2 – 10
Configuration Complexity
Linux has become incredibly configurable
Complexity increases considerably
; Source of bugs!
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)2 – 10
Configuration Complexity
Linux has become incredibly configurable
Complexity increases considerably
; Source of bugs!
Linux v3.0 contains:
7.702 Features
893 Kconfig files
31.281 Source files
88.897 #ifdef blocks
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)2 – 10
Finding Bugs with Tools for Static Analysis
Bugs in declaration and implementation
Excellent tool support for static analysis:
Coccinelle: Faults in Linux: Ten Years Later (ASPLOS’11)
Dingo: Taming Device Drivers (EuroSys’09)
KLEE: Automatic generation of high-coverage tests (EuroSys’08)
RWset: Attacking path explosion (TACAS’08)
EXE: Automatically generating inputs of death (CCS’06)
...
Each of them checks a single configuration:
What does allyesconfig actually cover?
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)3 – 10
Finding Bugs with Tools for Static Analysis
Bugs in declaration and implementation
Excellent tool support for static analysis:
Coccinelle: Faults in Linux: Ten Years Later (ASPLOS’11)
Dingo: Taming Device Drivers (EuroSys’09)
KLEE: Automatic generation of high-coverage tests (EuroSys’08)
RWset: Attacking path explosion (TACAS’08)
EXE: Automatically generating inputs of death (CCS’06)
...
Each of them checks a single configuration:
What does allyesconfig actually cover?
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)3 – 10
Finding Bugs with Tools for Static Analysis
Bugs in declaration and implementation
Excellent tool support for static analysis:
Coccinelle: Faults in Linux: Ten Years Later (ASPLOS’11)
Dingo: Taming Device Drivers (EuroSys’09)
KLEE: Automatic generation of high-coverage tests (EuroSys’08)
RWset: Attacking path explosion (TACAS’08)
EXE: Automatically generating inputs of death (CCS’06)
...
Each of them checks a single configuration:
What does allyesconfig actually cover?
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)3 – 10
Configuration Coverage
We define Configuration Coverage as:
fraction of selected configuration-conditional blocksdivided by the number of available configuration-conditional blocks.
How to catch bugs in obscure configurations?
Use a static analyzer that is able to detect the bug
Test each possible configuration individually
⇒ Not feasible
Static analyzers scan only a particular kernel configuration
⇒ How to effeciently extend their coverage?
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)4 – 10
Approach
MEMORY MODEL
FLATMEM
DISCONTIGMEM
SPARSEMEM NUMA
depends on
#ifdef CONFIG DISCONTIGMEM
// Block1
static . . . int pfn_to_mid(. . .)
# ifdef CONFIG NUMA
// Block2
# else
// Block3
# endif
#endif
Analysis of both Model and Implementation
Extraction of logical constraints
Find set of configuration that maximizes coverage
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)5 – 10
Approach
MEMORY MODEL
FLATMEM
DISCONTIGMEM
SPARSEMEM NUMA
depends on
#ifdef CONFIG DISCONTIGMEM
// Block1
static . . . int pfn_to_mid(. . .)
# ifdef CONFIG NUMA
// Block2
# else
// Block3
# endif
#endif
Analysis of both Model and Implementation
Extraction of logical constraints
Find set of configuration that maximizes coverage
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)5 – 10
Solution Outline
Build Variability Model
Reuse of the Implementation Variability Model [GPCE’10]
Additional constraints from Kconfig [Eurosys’11]
Use SAT solvers to find a “good” inital solution
Repeat until finished:
find a solution that adds additional blocksstore this selection of features as partial configuration
Caveat: Terminate even with dead blocks
Scan the source code with the resulting set of valid configurations
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)6 – 10
Implementation for Linux
KConfigFiles
config HOTPLUG_CPU bool "Support for ..." depends on SMP && ...
undertaker
Dead BlockDetection
KConfigParser
#ifdef CONFIG_HOTPLUG_CPU...#endif
Linuxsource file
CalculatePartialConfigs
ExpandPartialConfigs
Buildand
Testing
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)7 – 10
Implementation Challenges
Proper extraction of Configurations constraints
Kconfig (implemented in undertaker)Kbuild constraints (largely unhandled)
Expansion of Partial Configurations (Kconfig Fragments)
Naıve approach has some surprising effects (i.e., fails sometimes)Kconfig-sat seems promising, but unfortunately discontinued
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)8 – 10
Evaluation with Linux 2.6.35
Analyzed files 10,365Files with variability 3,163Rate of files with variability 30.52%Sum of all (partial) configurations 4,435
Sum of configuration controlled conditional blocks 16,444Sum of blocks selected by allyesconfig 11,511Sum of all blocks selected by undertaker-coverage 13,844Coverage allyesconfig (non-dead-corrected) 70.00%Coverage undertaker (non-dead-corrected) 84.19%
Dead blocks 1,778Selectable blocks (excluding dead blocks) 14,666Selected by allyesconfig 11,511Covered by undertaker 13,844allyesconfig coverage (dead-corrected) 78.49%undertaker coverage (dead-corrected) 94.40%undertaker coverage / allyesconfig coverage 1.20
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)9 – 10
Evaluation with Linux 2.6.35
Analyzed files 10,365Files with variability 3,163Rate of files with variability 30.52%Sum of all (partial) configurations 4,435Sum of configuration controlled conditional blocks 16,444Sum of blocks selected by allyesconfig 11,511Sum of all blocks selected by undertaker-coverage 13,844Coverage allyesconfig (non-dead-corrected) 70.00%Coverage undertaker (non-dead-corrected) 84.19%
Dead blocks 1,778Selectable blocks (excluding dead blocks) 14,666Selected by allyesconfig 11,511Covered by undertaker 13,844allyesconfig coverage (dead-corrected) 78.49%undertaker coverage (dead-corrected) 94.40%undertaker coverage / allyesconfig coverage 1.20
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)9 – 10
Evaluation with Linux 2.6.35
Analyzed files 10,365Files with variability 3,163Rate of files with variability 30.52%Sum of all (partial) configurations 4,435Sum of configuration controlled conditional blocks 16,444Sum of blocks selected by allyesconfig 11,511Sum of all blocks selected by undertaker-coverage 13,844Coverage allyesconfig (non-dead-corrected) 70.00%Coverage undertaker (non-dead-corrected) 84.19%Dead blocks 1,778Selectable blocks (excluding dead blocks) 14,666Selected by allyesconfig 11,511Covered by undertaker 13,844allyesconfig coverage (dead-corrected) 78.49%undertaker coverage (dead-corrected) 94.40%undertaker coverage / allyesconfig coverage 1.20
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)9 – 10
Evaluation with Linux 2.6.35
Analyzed files 10,365Files with variability 3,163Rate of files with variability 30.52%Sum of all (partial) configurations 4,435Sum of configuration controlled conditional blocks 16,444Sum of blocks selected by allyesconfig 11,511Sum of all blocks selected by undertaker-coverage 13,844Coverage allyesconfig (non-dead-corrected) 70.00%Coverage undertaker (non-dead-corrected) 84.19%Dead blocks 1,778Selectable blocks (excluding dead blocks) 14,666Selected by allyesconfig 11,511Covered by undertaker 13,844allyesconfig coverage (dead-corrected) 78.49%undertaker coverage (dead-corrected) 94.40%undertaker coverage / allyesconfig coverage 1.20
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)9 – 10
Evaluation with Linux 2.6.35
Analyzed files 10,365Files with variability 3,163Rate of files with variability 30.52%Sum of all (partial) configurations 4,435Sum of configuration controlled conditional blocks 16,444Sum of blocks selected by allyesconfig 11,511Sum of all blocks selected by undertaker-coverage 13,844Coverage allyesconfig (non-dead-corrected) 70.00%Coverage undertaker (non-dead-corrected) 84.19%Dead blocks 1,778Selectable blocks (excluding dead blocks) 14,666Selected by allyesconfig 11,511Covered by undertaker 13,844allyesconfig coverage (dead-corrected) 78.49%undertaker coverage (dead-corrected) 94.40%undertaker coverage / allyesconfig coverage 1.20
With 30 percent more static analysis runs(compiler calls)
→ 15 percent more Configuration Coverage
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)9 – 10
Conclusions
Configurability has to be seen as a significant cause ofsoftware defects in its own respect
The C Preprocessor imposes challenges for existing static analyzers
Configuration Coverage makes existing tools more effective
Further Discussions Points:
How accurate and useful is the current metric?Suggestions for other static analyzers?Analyze non-Linux projects
http://vamos.informatik.uni-erlangen.de/trac/undertaker
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)10 – 10
Conclusions
Configurability has to be seen as a significant cause ofsoftware defects in its own respect
The C Preprocessor imposes challenges for existing static analyzers
Configuration Coverage makes existing tools more effective
Further Discussions Points:
How accurate and useful is the current metric?Suggestions for other static analyzers?Analyze non-Linux projects
http://vamos.informatik.uni-erlangen.de/trac/undertaker
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)10 – 10
Conclusions
Configurability has to be seen as a significant cause ofsoftware defects in its own respect
The C Preprocessor imposes challenges for existing static analyzers
Configuration Coverage makes existing tools more effective
Further Discussions Points:
How accurate and useful is the current metric?Suggestions for other static analyzers?Analyze non-Linux projects
http://vamos.informatik.uni-erlangen.de/trac/undertaker
R. Tartler Configuration Coverage in the Analysis of Large-Scale System Software (October 23, 2011)10 – 10