Performance Debugging in the Large via Mining Millions of Stack Traces

Embedded Lab.Park Yeongseong

ICSE 2012Shi Han, Yingnong Dang, Song Ge, Dongmei Zhang, Microsoft Research Asia

Introduction Problem Formulation Approach Overview AOI(Area Of Interest) Extraction CallStack Pattern Mining CallStack Pattern Clustering Evaluation Conclusion Q&A

Contents

Modern software system◦ Large number of components◦ Large number of lines of code◦ Depend on large number of system components

Recent Solution◦ Microsoft Windows Error Reporting (WER)◦ Performance bugs not handled (crash/hang debug)

PerfTrack : Event Tracing for Windows Dtrace : Solaris and several other Unix-like sys-

tem

Introduction

Propose approach◦ StackMine

StackMine◦ To enable performance debugging in the large◦ Support scalable system for postmortem performance

debugging

StackMine include 2phases◦ Costly-pattern mining algorithm◦ Clustering on the mined costly pattern

Introduction

Problem Formulation Majority performance bug

◦ CPU consumption bug◦ Wait bug

Problem Formulation :Define some concepts

Input◦ A set of trace streams {}.◦ A number as the threshold of waiting time or CPU consumption

time. Output

◦ All callstack patterns for where , sorted by their costs in the descending order.

Problem Formulation : Basic-Problem Definition

3 steps to reduce the investigation scope◦ AOI extraction◦ Costly-maximal-pattern mining◦ CallStack pattern clustering

Approach Overview

Two major issues◦ Effectiveness◦ Efficiency

Two effective AOI extraction techniqes◦ Scope-based extraction◦ Content-based extraction

AOI(Area of Interest) Extraction

Scope-based extraction

Thread within a time period of bad performance.

◦ Wait Graph model

Content-based extraction

AOI(Area of Interest) Extraction

BIDE algorithm

Two major modification◦ modify the support of a subsequence pattern

from its occurrences to .◦ Maximal pattern checking : forward/backward

extension check.

CallStack Pattern Mining

More easily recognize the common part and the variant part of the callstack patterns.

Using performance metrics of a cluster can help produce better prioritization of results for investigation

CallStack Pattern Clustering

CallStack Pattern Clustering :Alignment

Match◦ Operation cost : 0

Insertion/deletion◦ Operation cost : 1.0

Substitution

CallStack Pattern Clustering :Similarity calculation

Window Explorer UI PerfTrack : collected 6000 trace stream Randomly select 1000 trace stream Relevant 921 trace stream( 181million call-

stack)◦ 140 million : waiting stacks◦ 41 running stacks

Evaluation

StackMine : ranked list 1215 pattern clus-ters.

AOI extract : 141million -> 689thousand Maximal-callstack-pattern mining :

produce 2239 costly patterns. Final Ranked costly pattern lists : 1215 Top 400 clusters

-> 93 performance signature 58.26% of the response delay time. Average 1.6 seconds of UI response delay

Evaluation

To enable performance debugging in the large : StackMine

StackMine helps performance analysts ef -fectively discover highly impactful perfor-mance bugs.

StackMine’s substantial benefits in perfor-mance debugging in the large

Conclusion

Q&A