Upload
victor-haynes
View
219
Download
1
Embed Size (px)
Citation preview
Automatically Inferring Temporal Properties for Program Evolution
Jinlin Yang and David Evans15th IEEE International Symposium on Software Reliability Engineering 5 November 2004Saint-Malo, FranceUniversity of
VirginiaComputer
Science
www.cs.virginia.edu/ipa 2
Temporal Properties• Constrain ordering of events
– Essential for program correctness (file open before read)
• Reveal important differences between programs: S P not a chef
• But, hard for humans to document correctly [Holzmann’s FSE 2002 keynote]
• Can we infer useful temporal properties automatically?This talk argues: Yes!
www.cs.virginia.edu/ipa 3
Dynamically Inferring Properties
• Inherently unsound: guessing properties of all executions by looking at a small number
• Value-based invariants– Daikon [Ernst, TSE, Feb 01]
• Temporal properties– Mining specification [Ammons, POPL 02]– Extracting component interfaces [Whaley, ISSTA
02]
• We focus on relationship between 2 or 3 events: automation, scalability
www.cs.virginia.edu/ipa 4
System Overview
Inferred Properties 1
DifferentProperties
Diff
ere
nce A
naly
zer
Inferred Properties 2
SharedPropertiesProgram
Version 1
Program Version 2
Dyn
am
ic In
fere
nce
… …
www.cs.virginia.edu/ipa 5
Property Inference
Inferred Properties 1
DifferentProperties
Diff
ere
nce A
naly
zer
Inferred Properties 2
SharedPropertiesProgram
Version 1
Program Version 2
Dyn
am
ic In
fere
nce
… …
ProgramInstrumented
Program
Instru
menta
tion
Test Suite
Execution Traces
Testin
g
Inferred Properties
CandidatePropertyPatterns
Infe
ren
ce
www.cs.virginia.edu/ipa 6
Candidate Property Patterns
• Response pattern [Dwyer, ICSE ‘99]
[-P]* (P [-S]* S [-P]*)*– Filtered version (all non P/S events
removed): S*(P+S+)*
SPPSPS SPSP • Too weak for our purposes
www.cs.virginia.edu/ipa 7
Partial Order of Patterns
MultiEffect
(PS+)*
MultiCause
(P+S)*
EffectFirst
S*(PS)*
Alternating
(PS)*
OneCause
S*(PS+)*
OneEffect
S*(P+S)*
CauseFirst
(P+S+)*
ResponseS*(P+S+)*
Str
icte
r
ΛΛΛ
Λ
3 P
rimitiv
ePatte
rns
4 D
eriv
ed
Patte
rns
www.cs.virginia.edu/ipa 8
Inferring Properties
• For all pairs of events, P and S:– Check which (if any) of
OneCause, CauseFirst and OneEffect are satisfied
– Infer the strictest property satisfied if more than one primitive is
MultiEffect(PS+)*
MultiCause
(P+S)*
EffectFirst
S*(PS)*
Alternating
(PS)*
OneCauseS*(PS+)*
OneEffect
S*(P+S)*
CauseFirst(P+S+)*
ΛΛΛ
Λ
www.cs.virginia.edu/ipa 9
Traces Example
Trace 1: PSPS Trace 2: PPS All Traces
CauseFirst
(P+S+)*
OneCauseS*(PS+)*
OneEffectS*(P+S)*
For any two events, determine the strictest pattern satisfied by all traces
www.cs.virginia.edu/ipa 10
Traces Example
Trace 1: PSPS Trace 2: PPS All Traces
CauseFirst
(P+S+)*
OneCauseS*(PS+)*
OneEffectS*(P+S)*
For any two events, determine the strictest pattern satisfied by all traces
www.cs.virginia.edu/ipa 11
Traces Example
Trace 1: PSPS Trace 2: PPS All Traces
CauseFirst
(P+S+)*
OneCauseS*(PS+)*
OneEffectS*(P+S)*
For any two events, determine the strictest pattern satisfied by all traces
CauseFirst OneEffect MultiCause
www.cs.virginia.edu/ipa 12
Implementation
ProgramInstrumented
Program
Instru
menta
tion
Test Suite
Execution Traces
Testin
g
Inferred Properties
CandidatePropertyPatterns
Infe
ren
ce
Automated for Java using JRat Method entry/exit eventsManually for C code or output (today’s examples)
Prototype (described in paper) 900 lines of PerlNew implementation 8K lines of Java Improved performance Analyses, ranking heuristics
www.cs.virginia.edu/ipa 13
Experiments
• Hypotheses– We can automatically extract interesting
temporal properties– Differences in inferred temporal properties
among multiple versions can reveal interesting things
• Target programs– Tour bus simulator (8 student submissions)– OpenSSL (0.9.6, 0.9.7-0.9.7d)
www.cs.virginia.edu/ipa 14
Tour Bus Simulator
• Bus and each passenger are a separate thread
• Assignment in Fall 2003 graduate-level course (before we started this project)
• 8 submissions from the instructor (all believed to be correct)
www.cs.virginia.edu/ipa 15
Testing
> cville_bus –N 2 –C 1 –T 2Bus waiting for trip 1Passenger 0 boardsBus drives around CharlottesvillePassenger 0 exitsBus waiting for trip 2Passenger 1 boardsBus drives around CharlottesvillePassenger 1 exitsBus stops for the day
N, the number of peopleC, the capacity of the busT, the number of trips
Executed each submission with 100 randomly generated inputs, where
20 < C ≤ 40C+1 ≤ N ≤ 2C1 ≤ T ≤ 10
Event traces extracted fromprogram output (no need to instrument program)
www.cs.virginia.edu/ipa 16
Differences Reveal Problems
Pattern7 Correct Versions
1 Faulty Version
Alternating
waitdrives
MultiEffect
drivesboardswaitexitswaitboards
waitdrives
waitboards
MultiCause
boardsdrives boardsdrives
CauseFirst
boardsexits boardsexitsdrivesexitswaitexits
– waitdrives not Alternating• Bus drives around before all passengers exit• Bug in locking code (misplaced synchronization)
– drivesexits, waitexits not MultiEffect• Bus drives around twice before letting passengers exit• Missing synchronization
www.cs.virginia.edu/ipa 17
OpenSSL
• Widely used implementation of the Secure Socket Layer protocol
• 6 versions [0.9.6, 0.9.7, 0.9.7a-d] between Sept 2000 and March 2004
• We focus on the handshake protocol– 38 different event types
www.cs.virginia.edu/ipa 18
SR_CLNT_HELLO SW_SRVR_HELLO
SW_CERT SW_KEY_EXCH SW_CERT_REQ
SW_SRVR_DONE
SR_CERT SR_KEY_EXCH
SR_CERT_VRFY SR_FINISHED
SW_CHANGE SW_FINISHED
BEFORE+ACCEPT
SW_FLUSH OK
SW_FLUSH
Client Server
www.cs.virginia.edu/ipa 19
Testing
• Manually instrumented server to record handshake events
• Executed each version of server with 1000 randomly generated clients– Client modified advance to a
randomly selected state with 5% probability
www.cs.virginia.edu/ipa 20
Inferred Alternating Patterns
0.9.6 0.9.7 0.9.7a
0.9.7b
0.9.7c
0.9.7d
SR_KEY_EXCHSR_CERT_VRFY
SW_CERTSW_KEY_EXCH
SW_SRVR_DONESR_CERT
Documented change:
ignore unrequeste
d client certificates
Fixed bug
causing server
crashes
Race condition
(present in all
versions)7 alternating patterns same for all versions
www.cs.virginia.edu/ipa 21
Partitioning Traces
All Traces
Correct Clients (never jump to random state) Faulty Clients
No Server Error Server Error
Follows SSL specification
Missing Alternating patterns:allows handshake cycle
Segmentation fault (<0.9.7d)
www.cs.virginia.edu/ipa 22
Performance• Prototype implementation (described in
paper)– Up to 3-4 minutes for examples (Pentium 4, 3GHz,
1G RAM)– Too slow (several days) for larger programs– Scales approximately as trace length events2
• New implementation– Scales approximately as trace length events– Analyze these experiments in < .5s
• Bus: 102 events, 100 traces, 222 events/trace: 0.4s• OpenSSL: 35 events, 1000 traces, 18 events/trace: 0.3s
– Feasible to analyze large systems• Thousands of different events• Traces with millions of events• Preliminary results from experiments with JBoss
www.cs.virginia.edu/ipa 23
Summary of Experiments• Useful in program evolution
– Reveal interesting changes in OpenSSL– Identify unexpected differences
• Revealed bugs– Tour bus: identified faulty implementation
• Multiple implementations are rare, but multiple representations are common (i.e. design, model, code)
– OpenSSL: • Differences between versions revealed• Flaws found by partitioning traces
www.cs.virginia.edu/ipa 24
Future Work/Research Questions
• Can we make this feasible in practice?– Automatically identify interesting events– Heuristics to identify important patterns– Scalability and automation
• Can we make it more useful?– More expressive patterns: more events,
combine with data-flow– Understand impact of different testing
strategies
www.cs.virginia.edu/ipa 25
Conclusion
• Automatically inferring temporal properties is feasible
• Even very simple property patterns reveal interesting program properties
www.cs.virginia.edu/ipa 26
Questions?
http://www.cs.virginia.edu/ipa
This work is funded in part by the National Science Foundation.
Thanks: Marty Humphrey for providing the student submissions, Chengdu Huang for help with OpenSSL, Joel Winstead, the anonymous reviewers, and anonymous CS650 students for writing buggy code.