Algorithm Portfolios for Noisy Optimization: Compare Solvers Early (LION8)

AlgorithmPortfolios for

NoisyOptimization:

CompareSolvers Early

Marie-LiesseCAUWET,Jialin LIU,Olivier

TEYTAUD

Outline

Black-boxNoisyOptimizationFramework

AlgorithmPortfolios

NoisyOptimizationAlgorithms(NOAs)

Experiments

Conclusions

References

Algorithm Portfolios for Noisy Optimization:Compare Solvers Early

TAO TeamINRIA Saclay-LRI-CNRS, Univ. Paris-Sud

91190 Gif-sur-Yvette, France

Marie-Liesse CAUWET Jialin LIU Olivier TEYTAUD

February 2014


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Introduction

Usually:

Portfolio of algorithms ! Combinatorial Optimization (C.O.)

New:

Portfolio of algorithms ! Noisy Optimization (N.O.)


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

1 Black-box Noisy Optimization Framework

2 Algorithm Portfolios

3 Noisy Optimization Algorithms (NOAs)

4 Experiments

5 Conclusions

6 References


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Black-box1 Noisy Optimization Framework

Let f = f (x ,!) from a domain D 2 Rd to R with ! randomvariable. We wish to find:

argminx

E!f (x ,!)

We have access to independent evaluations of f .

Notation: f (x) refers to f (x ,!).

1Black-box: we have no knowledge about the noise.


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Black-box Noisy Optimization Framework

Stochastic problem;

limited budget (here: total number of evaluations);

target: anytime convergence to the optimum;

black-box.

2

How to choose a suitable solver/optimizer?

2Image from

http://ethanclements.blogspot.fr/2010/12/postmodernism-essay-

question.html


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References


Stochastic problem;



black-box.

2


2Image from


question.html


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References


Stochastic problem;



black-box.

2


2Image from


question.html


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References


Stochastic problem;



black-box.

2


2Image from


question.html


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References


Stochastic problem;



black-box.

Algorithm Portfolios


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References


A finite number of given noisy optimization solvers,“orthogonal”;

distribution of budget;

information sharing.

! Performs almost as well as the best solver


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References


A finite number of given noisy optimization solvers,“orthogonal”;

distribution of budget;

information sharing.

! Performs almost as well as the best solver


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References


Algorithm 1 Noisy Optimization Portfolio Algorithm (NOPA).

1: Parameters: a dimension d 2 N⇤

2: Initialization: initialize a portfolio {S1, . . . , SM

} containing M solvers3: m, n 14: while (true) do5: for i = 1 to M do I Fair budget distribution6: Apply an iteration of solver S

i

until it has received at least n data samples7: x

i,n the current recommendation by solver Si

8: end for9: if n = r

m

then I Periodically we compare10: for i = 1 to M do11: Perform s

m

evaluations of the (stochastic) reward R(xi,k

n

)12: y

i

the average reward13: end for14: i

⇤ arg mini2{1,...,M}

y

i

I Who is best ?

15: m m + 116: end if17: x

n

x

i

⇤,n I Recommendation follows i

⇤

18: n n + 119: end while


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References


Algorithm 1 Noisy Optimization Portfolio Algorithm (NOPA).1: Parameters: a dimension d 2 N⇤


} containing M solvers3: m, n 1

4: while (true) do5: for i = 1 to M do I Fair budget distribution6: Apply an iteration of solver S

i




m


m


n

)12: y

i



y

i

I Who is best ?

15: m m + 116: end if17: x

n

x

i


⇤



NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References





i



8: end for

9: if n = r

m


m


n

)12: y

i



y

i

I Who is best ?

15: m m + 116: end if17: x

n

x

i


⇤



NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References





i




m


m


n

)12: y

i

the average reward13: end for

14: i


y

i

I Who is best ?

15: m m + 116: end if17: x

n

x

i


⇤



NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References





i




m


m


n

)12: y

i



y

i

I Who is best ?

15: m m + 116: end if

17: x

n

x

i


⇤



NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References





i




m


m


n

)12: y

i



y

i

I Who is best ?

15: m m + 116: end if17: x

n

x

i


⇤



NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References





i




m


m


n

)12: y

i



y

i

I Who is best ?

15: m m + 116: end if17: x

n

x

i


⇤



NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References


Compare Solvers Early

k

n

n: lag

8i 2 {1, . . . ,M}, xi ,k

n

6= or = x

i ,n

Why this lag ?comparing good points! comparing points with similar fitnesscomparing points with similar fitness! very expensivealgorithms’ ranking is usually stable! no use comparing the very last


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References



k

n

n: lag

8i 2 {1, . . . ,M}, xi ,k

n

6= or = x

i ,n



NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References



k

n

n: lag

8i 2 {1, . . . ,M}, xi ,k

n

6= or = x

i ,n

Why this lag ?

comparing good points! comparing points with similar fitnesscomparing points with similar fitness! very expensivealgorithms’ ranking is usually stable! no use comparing the very last


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References



k

n

n: lag

8i 2 {1, . . . ,M}, xi ,k

n

6= or = x

i ,n



NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Noisy Optimization Algorithms (NOAs)

SA-ES: Self-Adaptive Evolution Strategy;

Fabian’s algorithm: a first-order method using gradientsestimated by finite di↵erences[3, 2];

Noisy Newton’s algorithm: a second-order method using aHessian matrix approximated also by finite di↵erences[1];

. . .


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References





. . .


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References





. . .


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References





. . .


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

NOA 1: SA-ES with revaluations

Algorithm 2 Self-Adaptive Evolution Strategy with revaluations.1: Parameters: K > 0, ⇣ � 0, � � µ 2 N⇤, a dimension d 2 N⇤

2: Input: an initial parent x1,i 2 Rd and an initial �1,i = 1, i 2 {1, . . . , µ}3: n 14: while (true) do5: Generate � individuals i

j

, j 2 {1, . . . ,�}, independently usingI Generation

�j

= �n,mod(j�1,µ)+1 ⇥ exp

✓1

2dN

◆and i

j

= x

n,mod(j�1,µ)+1 + �j

N

6: Evaluate each of them dKn⇣e times and average their fitness valuesI Evaluation

7: Define j1, . . . , j� so that3 I Ranking

EdKn⇣e[f (ij1 )] EdKn⇣e[f (ij2 )] · · · EdKn⇣e[f (ij� )]

8: �n+1,k = �

j

k

and x

n+1,k = i

j

k

, k 2 {1, . . . , µ} I Updating9: n n + 1

10: end while

3Em

denotes the average over m resamplings


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

NOA 2: Fabian’s Algorithm

Algorithm 3 Fabian’s stochastic gradient algorithm with finitedi↵erences[5, 2].1: Parameters: a dimension d 2 N⇤, 1

2 > � > 0, a > 0, c > 0, m 2 N⇤, weightsw1 > · · · > w

m

summing to 1, scales 1 � u1 > · · · > u

m

> 02: Input: an initial x1 2 Rd

3: n 14: while (true) do5: Compute �

n

= c/n�

6: Evaluate the gradient g at xn

by finite di↵erences, averaging over 2m sam-ples per axis. 8i 2 {1, . . . , d}, 8j{1 . . .m}

x

(i,j)+n

= x

n

+ u

j

e

i

and x

(i,j)�n

= x

n

� u

j

e

i

g

i

=1

2�n

mX

j=1

w

j

⇣f (x(i,j)+

n

)� f (x(i,j)�n

)⌘

7: Gradient step: Apply x

n+1 = x

n

� a

n

g



NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

NOA 3: Noisy Newton’s algorithm

Algorithm 4 Noisy Newton’s algorithm with gradient and Hes-sian approximated by finite di↵erences and revaluations[1].1: Parameters: a dimension d 2 N⇤, A > 0, B > 0, ↵ > 0, � > 0, ✏ > 02: Input: h identity matrix, an initial x1 2 Rd

3: n 14: while (true) do5: Compute sigma

n

= A/n↵

6: Evaluate the gradient g at x

n

by finite di↵erences, averaging over dBn�esamples at distance ⇥(�

n

) of xn

7: for i = 1 to d do8: Evaluate Hessian h

i,i by finite di↵erences at x

n

+ �ei

and x

n

� �ei

,averaging each evaluation over dBn�e resamplings

9: for j = 1 to d do10: if i == j then11: Update h

i,j using h

i,i = (1� ✏)hi,i + ✏h

i,i

12: else13: Evaluate h

i,j by finite di↵erences thanks to evaluations at each ofx

n

± �ei

± �ej

, averaging over dBn�/10e samples

14: Update h

i,j using h

i,j = (1� ✏d

)hi,j +

✏d

h

i,j

15: end if16: end for17: end for


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

NOA 3: Noisy Newton’s algorithm

Algorithm 4 Noisy Newton’s algorithm with gradient and Hes-sian approximated by finite di↵erences and revaluations[1].

18: � solution of h� = �g I Newton step19: if � > C�

n

then20: � = C�

n

�||�||

21: end if22: Apply x

n+1 = x

n

+ �23: n n + 124: end while


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Experiments

A trivial problem

f (x) = ||x ||2 + ||x ||zN , x 2 Rd .

d : dimension;

N : a Gaussian standard noise;

z 2 {0, 1, 2}.


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Noisy optimization solvers

Table : Mono-solvers and portfolios used in the experiments.

Solvers Algorithm and parametrizationFabian1 Fabian’s solver with stepsize �

n

= 10/n0.1, a = 10.Fabian2 Fabian’s solver with stepsize �

n

= 10/n0.49, a = 100.Newton Newton’s solver with stepsize �

n

= 100/n4, resampling

n

= n

2.RSAES RSAES with � = 10d , µ = 5d , resampling

n

= 10n2.

Portfolio NOPA of 4 mono-solvers with k

n

= dn0.1e, rn

= n

3, sn

= 15n2.P.+ Sharing Portfolio with information sharing enabled.

Recall

n: portfolio iteration number;

r

n

: revaluation number for comparing at iteration n;

s

n

: comparison period;

k

n

: index of recommendation to be compared.


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Noisy optimization solvers

Table : Mono-solvers and portfolios used in the experiments.

Solvers Algorithm and parametrizationFabian1 Fabian’s solver with stepsize �

n

= 10/n0.1, a = 10.Fabian2 Fabian’s solver with stepsize �

n

= 10/n0.49, a = 100.Newton Newton’s solver with stepsize �

n

= 100/n4, resampling

n

= n

2.RSAES RSAES with � = 10d , µ = 5d , resampling

n

= 10n2.Portfolio NOPA of 4 mono-solvers with k

n

= dn0.1e, rn

= n

3, sn

= 15n2.P.+ Sharing Portfolio with information sharing enabled.

Recall

n: portfolio iteration number;

r

n

: revaluation number for comparing at iteration n;

s

n

: comparison period;

k

n

: index of recommendation to be compared.


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Simple Regret4[6]

For Simple Regret= SR

Let x⇤ be the optimum of f . Let xn

be the individual evaluatedat nth evaluation and x

n

the optimum estimated after nth

evaluation

Simple Regret SR = E(f (xn

)� f (x⇤))

Slope(SR) = limn!1

log(SR(n))

log(n)

4Di↵erence between average payo↵ recommended and optimal


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

f (x) = ||x ||2 + ||x ||zN in dimension 2

Solvers z = 0 z = 1 z = 2Fabian1 -1.24±0.05 -1.25±0.06 -1.23±0.06Fabian2 -0.17±0.09 -1.75±0.10 -3.16±0.06Newton -0.20±0.09 -1.84±0.34 -1.93±0.00RSAES -0.41±0.08 -0.61±0.13 -0.60±0.16Portfolio -1.00±0.28 -1.63±0.06 -2.69±0.07

P .+ Sharing -0.93±0.31 -1.64±0.05 -2.71±0.07

Table : Slope(SR) of experiments in dimension 2.

Best mono-solverWorst mono-solverPortfolios


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

f (x) = ||x ||2 + ||x ||zN in dimension 15

Solvers z = 0 z = 1 z = 2Fabian1 -0.83±0.02 -1.03±0.02 -1.02±0.02Fabian2 0.11±0.02 -1.30±0.02 -2.39±0.02Newton 0.00±0.02 -1.27±0.23 -1.33±0.00RSAES 0.15±0.01 0.14±0.02 0.15±0.01Portfolio -0.72±0.02 -1.06±0.01 -1.90±0.02

P .+ Sharing -0.72±0.02 -1.05±0.03 -1.90±0.03

Table : Slope(SR) of experiments in dimension 15.

Best mono-solverWorst mono-solverPortfolios


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Experiments

Results

The portfolio algorithm successfully reaches almost thesame slope(SR) as the best of its solvers;

for z = 2 the best algorithm is the second variant ofFabian’s algorithm;

for z = 1 the approximation of Noisy Newton’s algorithmperforms best;

for z = 0 the first variant of Fabian’s algorithm performsbest;

the sharing has little or no impact.


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Experiments

Results







NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Experiments

Results







NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Experiments

Results







NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Conclusions

Conclusions

Main conclusion:portfolios are classical in combinatorial optimization;(because in C.O. di↵erences between runtimes can behuge);portfolios also make a big di↵erence in noisy optimization;(because in N.O., with lag, comparison cost = small).

Sharing not that good.

We show mathematicallya and empirically a log(M) shiftwhen using M solvers, when working on a classical log-logscale (classical in noisy optimization).

A portfolio of solvers= approximately as e�cient as the bestb.

a

see paper :-)

b

More practical work can be found in [4].


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Conclusions

Conclusions

Main conclusion:portfolios are classical in combinatorial optimization;(because in C.O. di↵erences between runtimes can behuge);

portfolios also make a big di↵erence in noisy optimization;(because in N.O., with lag, comparison cost = small).




a

see paper :-)

b



NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Conclusions

Conclusions





a

see paper :-)

b



NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Conclusions

Conclusions





a

see paper :-)

b



NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Conclusions

Conclusions





a

see paper :-)

b



NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Conclusions

Conclusions





a

see paper :-)

b



NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Perspectives

Information sharing & unfair budget distribution

With 4 solvers, the log(M) shift is ok; with 40 maybe not.

Identifying relevant information for sharing.

If solver 1 says “I’ll never do better than X” and solver 2says “I have found at least Y > X” then we can stop 1.


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Perspectives






NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Perspectives






NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Some references

Sandra Astete-Morales, Marie-Liesse Cauwet, Jialin Liu, and OlivierTeytaud.Noisy optimization rates.submitted, 2013.

Vaclav Fabian.Stochastic Approximation of Minima with Improved Asymptotic Speed.Annals of Mathematical statistics, 38:191–200, 1967.

Jack Kiefer and Jacob Wolfowitz.Stochastic Estimation of the Maximum of a Regression Function.Annals of Mathematical statistics, 23:462–466, 1952.

Jialin Liu and Olivier Teytaud.Meta online learning: experiments on a unit commitment problem.In ESANN, Bruges, Belgium, 2014.

Ohad Shamir.On the complexity of bandit and derivative-free stochastic convexoptimization.CoRR, abs/1209.2388, 2012.

Gilles Stoltz, Sebastien Bubeck, and Remi Munos.Pure exploration in finitely-armed and continuous-armed bandits.Theoretical Computer Science, 412(19):1832–1852, April 2011.


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Thank you for your attention !

TAO Teamhttps://tao.lri.fr/tiki-index.php

INRIA Saclay-LRI-CNRS, Univ. Paris-SudDIGITEO, 91190 Gif-sur-Yvette, France

tao tao

tao tao

tao tao

MLC

tao tao

tao tao

tao tao

OT

https://tao.lri.fr/tiki-index.php


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Contacts

Algorithm Portfolios for Noisy Optimization:Compare Solvers Early

TAO Team, INRIA Saclay-LRI-CNRS, Univ. Paris-Sud

91190 Gif-sur-Yvette, France

Marie-Liesse CAUWET, Jialin LIU, Olivier TEYTAUD

Contacts:[email protected]

Personal page:https://www.lri.fr/⇠lastname/

Slides of presentation:https://www.lri.fr/⇠liu/portfolio2 lion8.pdf


NoisyOptimization:



TEYTAUD

Outline


AlgorithmPortfolios


Experiments

Conclusions

References

Rates Regret

For Regret = SR or CR

Slope(Regret) = limn!1

log(Regret(n))

log(n)

Algorithm Parameter Slope(SR) Slope(CR)

� ! 0 �1 1Fabian � ! 1

2 0 1� ! 1

4 �12

12

Presentations & Public Speaking

Algorithm Portfolios for Noisy Optimization: Compare Solvers Early (LION8)