Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with...

Mark HarmanFACEBOOK SOFTWARE ENGINEERING MANAGER

SAPIENZ TEAM; THIS TALK IS BASED ON WORK OF THE WHOLE TEAM

Mark HarmanFACEBOOK SOFTWARE ENGINEERING MANAGER

SAPIENZ TEAM; THIS TALK IS BASED ON WORK OF THE WHOLE TEAM

UNFORTUNATELY, YOUR APP HAS STOPPED.

OKREPORT

WORK IN PROGRESS

SLOW BORING UNIMPORTANTPAINFUL

URGENT

IN PROC. OF SSBSE, 2018.

Deploying Search Based Software Engineering with Sapienz at FacebookNADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO, ALEXANDER MOLS, TAIJIN TEI AND ILYA ZORIN

Search Based Optimization

Software Engineering

S B S E

We use Search Based Software Engineering

widely studied in academia; now starting to reach into industry

... covers history of SBST up to 2015

Test live in a search space

Engineers Design

Machines Execute

Unfortunately, your app has stopped.

System Level Testing

RANDOM FUZZER HUMAN TESTERS

Fault Triage Process

OPERATIONS IN THE EVOLUTION WORKFLOW

TRIAGEBUILD SEARCH CRASHREOCCURRING

OPERATOR

FILE A FIX STACK TRACE

A

B

C

DDIFFOWNER REVIEW LOCALIZER CRASH DATA

>1Msource control

commands run per day

>100Kcommits per week

Workplace for

Android

Messenger for

Android

Instagram for

Android

Facebook foriOS

Facebook for

Android

~75%FIX RATE

Distribution (FB)

Null Pointer

Illegal State

Illegal Argument

RunTime

Class Cast

No Such Method

Assertion

Array Index Out of Bounds

Bad Token

Index Out of Bounds

TOP CRASHES TYPES ON FACEBOOK FOR ANDROID (BY SAPIENZ)

Distribution (Research)

TOP CRASHES TYPES ON 1000 ANDROID APPS (BY SAPIENZ[1])

Null Pointer

Illegal State

Illegal Argument

RunTime

Activity Not Found

Out of Memory

Concurrent

Array Index Out of Bounds

Bad Token

Index Out of Bounds

[1] K. MAO, M. HARMAN, AND Y. JIA, “SAPIENZ: MULTI-OBJECTIVE AUTOMATED TESTING FOR ANDROID APPLICATIONS,” IN PROC. OF ISSTA’16, 2016

Distribution (Research)

TOP CRASHES TYPES ON 1000 ANDROID APPS (BY SAPIENZ[1])

Null Pointer

Illegal State

Illegal Argument

RunTime

Activity Not Found

Out of Memory

Concurrent

Array Index Out of Bounds

Bad Token

Index Out of Bounds

[1] K. MAO, M. HARMAN, AND Y. JIA, “SAPIENZ: MULTI-OBJECTIVE AUTOMATED TESTING FOR ANDROID APPLICATIONS,” IN PROC. OF ISSTA’16, 2016

Auto Fix

Auto BoostAuto Test Design Auto Fix

Auto Fix Workflow (Generation)

CRASH TRIGGERS

REVERT FULL DIFF

REVERTPARTIAL DIFF TEMPLATE MUTATION

SAPIENZ AUTOTRIAGE

FIX PATCH GENERATOR

TRIGGER PATCH GENERATOR

CREATE REVISION IN SAPFIX

Auto Fix Workflow (Validation)

CREATE REVISION IN SAPFIX

PATCH 1

PATCH 2

PATCH N

MANUALLY WRITTEN

TESTS

COMPILATION ERROR?BUILD

CRASH FIXED? NEW CRASH?SAPIENZ E2E

Auto Fix Workflow (Signal)

CREATE REVISION IN SAPFIX

REJECTED/ EXPRED

DEVELOPER REVIEWS

AUTO FIX PROCESSOR

DEVELOPER INTERACTION

TRACKER

ACCEPTED

SAPIENZ

INFER

Localization: Sapienz + Infer

SAPIENZ 1559

INFER

// Sapienz Revealed NPE Crash contextDateMap.put("name", displayedUser.getName());

1559// INFER WarningThe value of 'UserDetailFragment.displayedUser in the call to 'getName()'could be null.

Summary

AUTO TEST DESIGN

AUTO FIX AUTO BOOST

Flaky testsFix detectionAutomated OraclesWider search spacesSmarter white box coverageCombining static and dynamicUnit test from system testsHuman machine test hybridsFully parallel search algorithmsAuto Fix and perf improvement

IN PROC. OF SSBSE, 2018.

Deploying Search Based Software Engineering with Sapienz at FacebookNADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO, ALEXANDER MOLS, TAIJIN TEI AND ILYA ZORIN

Resources