Hyperparameter Optimization 101

Hyperparameter Optimization 101Alexandra Johnson

Software Engineer, SigOpt

What are Hyperparameters?

Hyperparameters affect model performance

How Do I Find The Best Hyperparameters?

Step 1: Pick an Objective Metric

Classification models Accuracy

Regression models Root MSE

Caveat: Cross Validate to Prevent Overfitting

Cross Validation

4 5 60 1 2 3 7 8 9

4 5 6 7 8 90 1 2 3data

train validate metric

Cross Validation

4 5 60 1 2 3 7 8 9

4 5 6 7 8 90 1 2 3data

train

6 7 91 2 4 5 0 3 8train

7 8 90 2 3 6 1 4 5train

metric

metric

metric

K ti

mes validate

validate

validate

Grid Search Random Search Bayesian Optimization

Step 2: Pick an Optimization Strategy

Step 3: Evaluate N Times

N Times

What is the Best Hyperparameter Optimization Strategy?

Primary Consideration: How Good are the “Best” Hyperparameters?

“Best Found Value” Distributionsex

perim

ents

accuracy

Secondary Consideration: How Much Time Do You Have?

Number of Evaluations Required

Grid Search Random Search Bayesian Optimization

2 parameters 100 ?? 20-40

3 parameters 1,000 ?? 30-60

4 parameters 10,000 ?? 40-80

5 parameters 100,000 ?? 50-100

SigOptEasy-to-use REST API, R, Java, Python Clients

Ensemble of Bayesian optimization techniques

Free trial, academic discount, we’re hiring!

SigOpt Tutorial VideosVersus untuned models:

+315.2% accuracy with TensorFlow CNN

+49.2% accuracy with Xgboost + unsupervised features

Learn MoreSee more at sigopt.com/research:● Blog posts● Papers● Videos

Thank You!Twitter: @SigOpt

Email: support@sigopt.com

Web: sigopt.com/getstarted