Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Mark HarmanFACEBOOK SOFTWARE ENGINEERING MANAGER
SAPIENZ TEAM; THIS TALK IS BASED ON WORK OF THE WHOLE TEAM
Mark HarmanFACEBOOK SOFTWARE ENGINEERING MANAGER
SAPIENZ TEAM; THIS TALK IS BASED ON WORK OF THE WHOLE TEAM
UNFORTUNATELY, YOUR APP HAS STOPPED.
OKREPORT
WORK IN PROGRESS
SLOW BORING UNIMPORTANTPAINFUL
URGENT
IN PROC. OF SSBSE, 2018.
Deploying Search Based Software Engineering with Sapienz at FacebookNADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO, ALEXANDER MOLS, TAIJIN TEI AND ILYA ZORIN
Search Based Optimization
Software Engineering
S B S E
We use Search Based Software Engineering
widely studied in academia; now starting to reach into industry
... covers history of SBST up to 2015
Test live in a search space
Engineers Design
Machines Execute
Unfortunately, your app has stopped.
System Level Testing
RANDOM FUZZER HUMAN TESTERS
Fault Triage Process
OPERATIONS IN THE EVOLUTION WORKFLOW
TRIAGEBUILD SEARCH CRASHREOCCURRING
OPERATOR
FILE A FIX STACK TRACE
A
B
C
DDIFFOWNER REVIEW LOCALIZER CRASH DATA
>1Msource control
commands run per day
>100Kcommits per week
Workplace for
Android
Messenger for
Android
Instagram for
Android
Facebook foriOS
Facebook for
Android
~75%FIX RATE
Distribution (FB)
Null Pointer
Illegal State
Illegal Argument
RunTime
Class Cast
No Such Method
Assertion
Array Index Out of Bounds
Bad Token
Index Out of Bounds
TOP CRASHES TYPES ON FACEBOOK FOR ANDROID (BY SAPIENZ)
Distribution (Research)
TOP CRASHES TYPES ON 1000 ANDROID APPS (BY SAPIENZ[1])
Null Pointer
Illegal State
Illegal Argument
RunTime
Activity Not Found
Out of Memory
Concurrent
Array Index Out of Bounds
Bad Token
Index Out of Bounds
[1] K. MAO, M. HARMAN, AND Y. JIA, “SAPIENZ: MULTI-OBJECTIVE AUTOMATED TESTING FOR ANDROID APPLICATIONS,” IN PROC. OF ISSTA’16, 2016
Distribution (Research)
TOP CRASHES TYPES ON 1000 ANDROID APPS (BY SAPIENZ[1])
Null Pointer
Illegal State
Illegal Argument
RunTime
Activity Not Found
Out of Memory
Concurrent
Array Index Out of Bounds
Bad Token
Index Out of Bounds
[1] K. MAO, M. HARMAN, AND Y. JIA, “SAPIENZ: MULTI-OBJECTIVE AUTOMATED TESTING FOR ANDROID APPLICATIONS,” IN PROC. OF ISSTA’16, 2016
Auto Fix
Auto BoostAuto Test Design Auto Fix
Auto Fix Workflow (Generation)
CRASH TRIGGERS
REVERT FULL DIFF
REVERTPARTIAL DIFF TEMPLATE MUTATION
SAPIENZ AUTOTRIAGE
FIX PATCH GENERATOR
TRIGGER PATCH GENERATOR
CREATE REVISION IN SAPFIX
Auto Fix Workflow (Validation)
CREATE REVISION IN SAPFIX
PATCH 1
PATCH 2
PATCH N
MANUALLY WRITTEN
TESTS
COMPILATION ERROR?BUILD
CRASH FIXED? NEW CRASH?SAPIENZ E2E
Auto Fix Workflow (Signal)
CREATE REVISION IN SAPFIX
REJECTED/ EXPRED
DEVELOPER REVIEWS
AUTO FIX PROCESSOR
DEVELOPER INTERACTION
TRACKER
ACCEPTED
SAPIENZ
INFER
Localization: Sapienz + Infer
SAPIENZ 1559
INFER
// Sapienz Revealed NPE Crash contextDateMap.put("name", displayedUser.getName());
1559// INFER WarningThe value of 'UserDetailFragment.displayedUser in the call to 'getName()'could be null.
Summary
AUTO TEST DESIGN
AUTO FIX AUTO BOOST
Flaky testsFix detectionAutomated OraclesWider search spacesSmarter white box coverageCombining static and dynamicUnit test from system testsHuman machine test hybridsFully parallel search algorithmsAuto Fix and perf improvement
IN PROC. OF SSBSE, 2018.
Deploying Search Based Software Engineering with Sapienz at FacebookNADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO, ALEXANDER MOLS, TAIJIN TEI AND ILYA ZORIN
Resources