D E C E M B E R 8 - 9 , 2 0 1 6

BigML, Inc 2

Poul Petersen CIO, BigML, Inc.

Feature EngineeringCreating Machine Learning Ready Data

BigML, Inc 3Feature Engineering

Machine Learning Secret

“…the largest improvements in accuracy often came from quick experiments, feature engineering, and model tuning rather than applying fundamentally different algorithms.”

Facebook FBLearner 2016

Feature Engineering: applying domain knowledge of the data to create features that make machine

learning algorithms work better or at all.

BigML, Inc 4Feature Engineering

Obstacles• Data Structure

• Scattered across systems • Wrong "shape" • Unlabelled data

• Data Value • Format: spelling, units • Missing values • Non-optimal correlation • Non-existant correlation

• Data Significance • Unwanted: PII, Non-Preferred • Expensive to collect • Insidious: Leakage, obviously correlated

Data Transformation

Feature Engineering

Feature Selection

BigML, Inc 5Feature Engineering

Feature Engineering

2013-09-25 10:02

Automatic Date Transformation

… year month day hour minute …

… 2013 Sep 25 10 2 …

… … … … … … …

NUM NUMCAT NUM NUM

DATE-TIME

BigML, Inc 6Feature Engineering

Feature EngineeringAutomatic Categorical Transformation

… alchemy_category …… business …… recreation …… health …… … …

CAT

business health recreation …… 1 0 0 …… 0 0 1 …… 0 1 0 …… … … … …

NUM NUM NUM

BigML, Inc 7Feature Engineering

Feature Engineering

Be not afraid of greatness: some are born great, some achieve greatness, and some have greatnessthrust upon ‘em.

TEXT

Automatic Text Transformation

… great afraid born achieve …

… 4 1 1 1 …

… … … … … …

NUM NUM NUM NUM

BigML, Inc 8Feature Engineering

Feature Engineering

{ “url":"cbsnews", "title":"Breaking News Headlines Business Entertainment World News “, "body":" news covering all the latest breaking national and world news headlines, including politics, sports, entertainment, business and more.”}

Fixing "non-optimal correlations"

title body

Breaking News… news covering…

… …

TEXT TEXT

TEXT

BigML, Inc 9Feature Engineering

Feature EngineeringDiscretization

Total Spend

7.342,99

304,12

4,56

345,87

8.546,32

NUM

“Predict will spend $3,521 with error

$1,232”

Spend Category

Top 33%

Bottom 33%

Bottom 33%

Middle 33%

Top 33%

CAT

“Predict customer will be Top 33% in

spending”

BigML, Inc 10Feature Engineering

Feature EngineeringCombinations of Multiple Features

Kg M2

101,4 3,24

85,2 2,8

56,2 2,9

136,1 3,6

95,9 4,1

NUM NUM

BMI

31,29

30,42

19,38

37,81

23,39

NUM

Kg M2

BigML, Inc 11Feature Engineering

Feature EngineeringFlatline

• BigML’s Domain-Specific Language (DSL) for Transforming Datasets

• Limited programming language structures

• let, cond, if, maps, list operators, */+-

• Dataset Fields are first-class citizens

• (field “diabetes pedigree”)

• Built-in transformations

• statistics, strings, timestamps, windows

BigML, Inc 12Basic Transformations

Data LabellingData may not have labels needed for doing classification

Create specific metrics for adding labels

Name Month - 3 Month - 2 Month - 1

Joe Schmo 123,23 0 0

Jane Plain 0 0 0

Mary Happy 0 55,22 243,33

Tom Thumb 12,34 8,34 14,56

Un-Labelled Data

Labelled dataName Month - 3 Month - 2 Month - 1 Default

Joe Schmo 123,23 0 0 FALSE

Jane Plain 0 0 0 TRUE

Mary Happy 0 55,22 243,33 FALSE

Tom Thumb 12,34 8,34 14,56 FALSE

(= 0 (+ (abs ( f "Month - 3" ) ) (abs ( f "Month - 2")) (abs ( f "Month - 1") ) ))

BigML, Inc 13Feature Engineering

Feature Engineering

(/ (- ( f "price") (avg-window "price" -4, -1)) (standard-deviation "price"))

date volume price1 34353 3142 44455 3153 22333 3154 52322 3215 28000 3206 31254 3197 56544 3238 44331 3249 81111 287

10 65422 29411 59999 30012 45556 30213 19899 30114 21453 302

day-4 day-3 day-2 day-1 4davg -

314 -314 315 -

314 315 315 -314 315 315 321 316,25315 315 321 320 317,75315 321 320 319 318,75

Current - (4-day avg) std dev

Shock: Deviations from a Trend

BigML, Inc 14Feature Engineering

Feature Engineering

(/ (- (f "price") (avg-window "price" -4, -1)) (standard-deviation "price"))

Current - (4-day avg) std dev

Shock: Deviations from a Trend

Current : (field “price”) 4-day avg: (avg-window “price” -4 -1) std dev: (standard-deviation “price”)

BigML, Inc 15Feature Engineering

Feature EngineeringMoon Phase%

( / ( mod ( - ( / ( epoch ( field {{date-field}} )) 1000 ) 621300 ) 2551443 ) 2551442 )

BigML, Inc 16Feature Engineering

Feature EngineeringFixing "non-existant correlations"

Highway Number Direction Is Long2 East-West FALSE4 East-West FALSE5 North-South TRUE8 East-West FALSE10 East-West TRUE… … …

Goal: Predict principle direction from highway number

( = (mod (field "Highway Number") 2) 0)

BigML, Inc 17Feature Engineering

Feature EngineeringFix Missing Values in a “Meaningful” Way

Filter Zeros

Model insulin

Predict insulin

Select insulin

FixedDataset

AmendedDataset

OriginalDataset

CleanDataset

( if ( = (field "insulin") 0) (field "predicted insulin") (field "insulin"))

BigML, Inc 18

Feature Selection

BigML, Inc 19Feature Engineering

Feature Selection• Model Summary

• Field Importance • Algorithmic

• Best-First Feature Selection • Boruta

• Leakage • Tight Correlations (AD, Plot, Correlations) • Test Data • Perfect future knowledge

cat diabetes.csv diabetes_testset.csv | sort | uniq -d | wc -l

BigML, Inc 20

Evaluate & Automate

BigML, Inc 21Feature Engineering

Evaluate & Automate

• Evaluate • Did you meet the goal? • If not, did you discover something else useful? • If not, start over • If you did…

• Automate - You don’t want to hand code that every time, right? • Consider tools that are easy to automate

• scripting interface • APIs • Ability to maintenance is important

BigML, Inc 22Feature Engineering

The Process

Data TransformDefine Goal Model &

Evaluate

no

yes

BetterData

NotPossible

TuneAlgorithm

Goal Met?

Automate

Feature Engineer & Selection

BetterFeatures