Vulnerability Prediction in Android Apps · Vulnerability Prediction in Android Apps Android apps are an attractive target et App security is not guaranteed by the platform provider!

Aram Hovsepyan1, Riccardo Scandariato1,James Walden2, Viet Hung Nguyen3

Wouter Joosen1

1 iMinds-DistriNet, Katholieke Universiteit Leuven2 Northern Kentucky University, 3 University of Trento

Vulnerability Prediction in Android Apps

Monday 10 June 13 week


Dis

triN

et

Android apps are an attractive target

Android has 75% market share as of Q1 2013 [IDC]

2Monday 10 June 13 week


Dis

triN

et


Google play has over 775K apps and over 48B total installs [IDC, Google I/O keynote]



Dis

triN

et


App security is not guaranteed by the platform providerÜ Apps that are well intended, but not exploit free

A single vulnerability could affect a massive number of users

Not yet much exploredÜ Focused on Mozilla Firefox / RHEL



Dis

triN

et

How to find vulnerabilities?



Dis

triN

et


Code inspectionÜ Manual verification is not feasible

Ü Not all apps can afford security experts

Ü Even security experts cannot analyze every line of code



Dis

triN

et





Penetration testing / security testing



Dis

triN

et






Static code analysis



Dis

triN

et






Static code analysis

MagicÜ Vulnerability prediction models



Dis

triN

et

Vulnerability prediction model



Dis

triN

et


6

Source code

Source code

Source code

Source code

Source code

Source code



Dis

triN

et


6

Source code

Source code

Source code

Source code

Source code

Source code

Source code

Source code

Source code

Source code

Source code

Source code



Dis

triN

et


6

Source code

Source code

Source code

Source code

Source code

Source code

Source code

Source code

Source code

Source code

Source code

Source code

Machine learning



Dis

triN

et

Our research

Predict vulnerable Java files in Android apps!

Predict vulnerable C++ components in Chrome/FirefoxÜ ongoing

Predict vulnerable PHP filesÜ summer work



Dis

triN

et

Outline

Existing tools and techniquesÜ Vulnerability prediction models

Our approach

Results

Conclusions and future research



Dis

triN

et

Vulnerability prediction models



Dis

triN

et


9

Source code

Source code

Source code

Source code

Source code

Source code

Source code

Source code

Source code

Source code

Source code

Source code



Dis

triN

et




Dis

triN

et


Start from a hunch = featureÜ e.g., larger components are more likely to be vulnerable



Dis

triN

et



Fetch the features from the componentsÜ e.g., calculate the size for each component



Dis

triN

et




Determine the vulnerabilitiesÜ e.g., National Vulnerability Database, MFSA



Dis

triN

et




Determine the vulnerabilitiesÜ e.g., National Vulnerability Database, MFSA

Investigate the correlationÜ Use machine learning techniques



Dis

triN

et




Dis

triN

et


Typical “hunches”Ü Use size and complexity metrics

Ü Leverage developer activity metrics

Ü Leverage code churn metrics

Ü Leverage design churn metrics

Ü Number of import statements



Dis

triN

et


Typical “hunches”Ü Use size and complexity metrics

Ü Leverage developer activity metrics

Ü Leverage code churn metrics

Ü Leverage design churn metrics

Ü Number of import statements

Inspired on the defect prediction workÜ Vulnerabilities are actually defects, but much more scarce

(“needle in a haystack”)



Dis

triN

et


The existing models are fairly complexÜ Typically several versions are necessary to collect all

metrics

Ü Developer activity metrics are required

Ü Code evolution metrics are required

Biased to the underlying “hunch” of the researcher



Dis

triN

et

Outline

Existing tools and techniquesÜ Static code analysis

Ü Vulnerability prediction using metrics

Our approach

Results




Dis

triN

et

Our approach

Use the source code itself in a tokenized form

Use the token frequency as featuresÜ Simplicity

Ü No explicit assumptions regarding the code characteristics

13

#machine learning

#SPAM filtering

#text analysis



Dis

triN

et

Our approach

14

Source code(Java files)

Tokenizer Feature vectors

Machine Learning

FortifyStatic code

analyzer

Vulnerabilities



Dis

triN

et

Tokenizer

15



Machine Learning

FortifyStatic code

analyzer

Vulnerabilities



Dis

triN

et

Tokenizer

15



Machine Learning

FortifyStatic code

analyzer

Vulnerabilities



Dis

triN

et

Tokenizer

Transform each source code token into a feature vectorÜ each token (“monogram”) is a feature

Ü tokenize by delimiters, mathematical and logical operations. , ! < > [ ] = + - ^ * / etc.

Ü each feature has a count assigned to it

15



Machine Learning

FortifyStatic code

analyzer

Vulnerabilities



Dis

triN

et

Feature vector



Dis

triN

et

Feature vector



Dis

triN

et

Feature vector

16

package: 1



Dis

triN

et

Feature vector

16

package: 1, com: 1package: 1



Dis

triN

et

Feature vector

16

package: 1, com: 1, fsck: 1, k9: 1, import: 2, android: 2, text: 2, util: 1, Rfc822Tokenizer: 2, widget: 1, AutoCompleteTextView:1, Validator: 2, public: 3, class: 1, EmailAddressValidator: 1, implements: 1, CharSequence: 2, fixText: 1, invalidText: 1, return: 2, tokenize: 1, length: 1

package: 1, com: 1package: 1



Dis

triN

et

Vulnerability assignment

17



Machine Learning

FortifyStatic code

analyzer

Vulnerabilities



Dis

triN

et


17



Machine Learning

FortifyStatic code

analyzer

Vulnerabilities



Dis

triN

et


Assign vulnerability to each Java fileÜ use Fortify (static code analyzer) for this task

Ü each file is either vulnerable or clean

17



Machine Learning

FortifyStatic code

analyzer

Vulnerabilities



Dis

triN

et




17



Machine Learning

FortifyStatic code

analyzer

Vulnerabilities




Dis

triN

et




17



Machine Learning

FortifyStatic code

analyzer

Vulnerabilities

package: 1, com: 1, fsck: 1, k9: 1, import: 2, android: 2, text: 2, util: 1, Rfc822Tokenizer: 2, widget: 1, AutoCompleteTextView:1, Validator: 2, public: 3, class: 1, EmailAddressValidator: 1, implements: 1, CharSequence: 2, fixText: 1, invalidText: 1, return: 2, tokenize: 1, length: 1, vulnerability: 0




Dis

triN

et

Machine learning

18



Machine Learning

FortifyStatic code

analyzer

Vulnerabilities



Dis

triN

et

Machine learning

18



Machine Learning

FortifyStatic code

analyzer

Vulnerabilities



Dis

triN

et

Machine learning

Leverage machine learning techniques to build a prediction modelÜ Training set -> the data used to train the model

Ü Testing set -> the data used to validate the model

Various techniques available (SVM, Naive Bayes, Random Forest, CART, kNN)

18



Machine Learning

FortifyStatic code

analyzer

Vulnerabilities



Dis

triN

et

Experiment 1



Dis

triN

et

Experiment 1

Can we predict future versions of an app based on its first version?



Dis

triN

et

Experiment 1

Can we predict future versions of an app based on its first version?Ü Training set - the first version (v0) of an app



Dis

triN

et

Experiment 1


Ü Testing set - all subsequent versions of that app



Dis

triN

et

Experiment 1





Dis

triN

et

Experiment 1





Dis

triN

et

Experiment 1



Ü Repeat for all apps



Dis

triN

et

Experiment 2



Dis

triN

et

Experiment 2

Can we build a generalized predictor that works on all apps?Ü Training set - the first version (v0) of an app

Ü Testing set - first versions of all other apps



Dis

triN

et

Applications (data from early 2012)

Ü F-droid repository: 01/01/2010->31/12/2011

Ü Selection criteria: open-source, size, number of versions



Dis

triN

et

Applications: descriptive statistics



Dis

triN

et

Applications: descriptive statistics



Dis

triN

et

Outline

Existing tools and techniquesÜ Static code analysis

Ü Vulnerability prediction using metrics

Our approach

Results




Dis

triN

et

Performance indicators



Dis

triN

et


Accuracy: percentage of correctly classified filesÜ imagine 90% of the files are clean

Ü saying all files are clean will achieve 90% accuracy



Dis

triN

et






Dis

triN

et






Dis

triN

et




Prediction vs. realityÜ True positive (TP)

Ü True negative (TN)

Ü False positive (FP)

Ü False negative (FN)



Dis

triN

et








25

TP

TN

TP

TP

FN

FNFN

FPTN

TN

TNFP



Dis

triN

et








Precision: P = TP/(TP+FP)

25

TP

TN

TP

TP

FN

FNFN

FPTN

TN

TNFP

P = 3/5



Dis

triN

et









25

TP

TN

TP

TP

FN

FNFN

FPTN

TN

TNFP



Dis

triN

et









Recall: R = TP/(TP+FN)

25

TP

TN

TP

TP

FN

FNFN

FPTN

TN

TNFP

R = 3/6



Dis

triN

et









Recall: R = TP/(TP+FN)

25

TP

TN

TP

TP

FN

FNFN

FPTN

TN

TNFP



Dis

triN

et

Performance indicators: K9



Dis

triN

et

Performance indicators: K9

26

0.00#

0.10#

0.20#

0.30#

0.40#

0.50#

0.60#

0.70#

0.80#

0.90#

1.00#

Mar010#

Apr010#

May010#

Jun010#

Jul010#

Aug010#

Sep010#

Oct010#

Nov010#

Dec010#

Jan011#

Feb011#

Mar011#

Apr011#

May011#

Jun011#

Jul011#

Aug011#

Sep011#

Oct011#

Nov011#

Dec011#

K9Mail#(Random#Forest)#

Precision#

Recall#



Dis

triN

et

Experiment 1: future predictions



Dis

triN

et


27

When do we need to build a new model?Ü Retrain when performance

indicators drop with 10%



Dis

triN

et


27

Application Retrain (months)AnkiDroid --BoardGameGeek 9ConnectBot --CoolReader 10Crosswords 2FBReader --K9Mail 12KeePassAndroid --MileageTracker 1Mustard --

-- no retraining is required

When do we need to build a new model?Ü Retrain when performance

indicators drop with 10%



Dis

triN

et

Results: most influential features



Dis

triN

et


Most influential features



Dis

triN

et


Most influential featuresÜ e, Exception, try, catch (error handling)



Dis

triN

et



Ü if (branching)



Dis

triN

et



Ü if (branching)

Ü null (pointer algebra)



Dis

triN

et



Ü if (branching)


Ü java, org (import statements)



Dis

triN

et



Ü if (branching)



Ü new, Log (others)



Dis

triN

et



Ü if (branching)



Ü new, Log (others)

Produced by InfoGain



Dis

triN

et

Validity threats

Use of Fortify tool for vulnerability extractionÜ Some research results have shown that there are strong

correlations between static analysis metrics and the quality of reported vulnerabilities

Ü Manual validation seems to confirm our findings (work in progress)!

Ü We are currently validating the same technique on Mozilla Firefox and the results are slightly better than the existing work



Dis

triN

et


We have presented a novel technique for predicting vulnerable Java files in Android applicationsÜ The obtained results are very promising

We are working in parallel on 2 additional tracksÜ Vulnerability prediction for Firefox/Chrome in C++

Ü Vulnerability prediction for PHP



Dis

triN

et

Bring your own data

We are looking to validate our technique further

If you have data you are willing to share with us, we would be glad to collaborate


Documents

Vulnerability Prediction in Android Apps · Vulnerability Prediction in Android Apps Android apps are an attractive target et App security is not guaranteed by the platform provider!