SCILAB WORKSHOP...approximation for pour Hessian 𝑘 • ρ𝑘 pour control convergence...

1www.esi-group.com

www.esi-group.com

SCILAB WORKSHOP

Optimization in SCILAB

SCILAB TEAM

17th, October 2019

2www.esi-group.com

Advanced post-processingAirfoil shape optimization

3www.esi-group.com

Quick introduction to Numerical Optimization

4www.esi-group.com

Optimization problem

max𝑥∊𝑋

𝑓 𝑥

3 Steps:

1/ Identification of the decision variables: x

2/ Set-up of constraints associated to the variables: X

3/ Definition of a cost/objective function: f

5www.esi-group.com

Two formulations, One problem

Solving the problem

min𝑥∊𝑋

𝑓(𝑥)

Corresponds to solve the problem

max𝑥∊𝑋

−𝑓 𝑥

Minimize or Maximize a function?

6www.esi-group.com

Numerical optimization

Initialization

Evaluation of cost

function

Choice of direction

New element

Optimal element

7www.esi-group.com

1/ Initial approximation– Influence convergence

2/ Number of iterations– Recursive process

3/ Convergence speed

4/ Stopping criteria

• Maximum number of iterations, …

• Cost function value under a given threshold, …

𝑓(𝑥𝑛) < 𝜀1

• Difference between two consecutive approximations under a giventhreshold, …

𝑥𝑛+1 − 𝑥𝑛 < 𝜀2

Classical set-up parameters for optimization solvers

8www.esi-group.com

General remarks

• Local extrema are easy to find!

• Some algorithm are performinglocal search only!

Take care on initial

approximation!

WARNING : Different types of extrema

9www.esi-group.com

Direct search

Search for an optimal position usingcost function evaluations, without anycomputation of the function exact orapproximate derivatives.

Interior Points method, simplex

Maximum search methods

Derivatives computation

Search for an optimal position usingcost function derivatives (Jacobian forsteepest ascent direction, Hessian forcritical point type) exact or approximatecomputation.

Gradient method, Newton method

How to make design space exploration?

10www.esi-group.com

Optimization problem typology

11www.esi-group.com

Travelling salesman problem

Continuous vs Discrete

DISCRETE (Combinatorial Analysis)

Variables are taking values in a finite setor states (IN, ON/OFF, …).

Classical examples: Travelling salesmanproblem which consists in finding theshortest path binding a given set of cities.

12www.esi-group.com

Mono vs Multi objective

MULTI OBJECTIVE

min𝑥∊𝑋

(𝑓1 𝑥 , 𝑓2 𝑥 ,… , 𝑓𝑘 𝑥 )

Several cost, potentially conflicting,functions minimization.

Found solution is therefore not unique.The complete set of solutions is calledPareto Front.

Choice is about trade-offAn example would be to ensuremaximum efficiency, and minimuminvestment at the same time.

13www.esi-group.com

Constrained vs Non Constrained

CONSTRAINTS

Constraints are referring to the limit of the design space we impose

• Linear

𝐶𝑥 = 𝑏

• Non Linear

𝑥𝑇𝐶𝑥 = 𝑏

• Equalities𝐶𝑥 = 𝑏

• Inequalities𝐶𝑥 ≥ 𝑏

• Lower and Upper bounds𝑎1 ≤ 𝑥 ≤ 𝑎2

14www.esi-group.com

Problem size

Let n be the number of design variables

• n < 10 : Small problem (s)

• 10 < n < 100 : Medium problem (m)

• 100 < n < 1000 : Large problem (l)

Solver speed and memory needsare depending on the problem size

15www.esi-group.com

Optimization solvers in SCILAB

16www.esi-group.com

How to choose SCILAB optimization solver

Convex

SemidefiniteLeast squares

Quadratic

Linear

Objective Constraints

[Bnds, Eq, Ineq]

Size Gradient Solver

Linear Y Y Y m karmarkar

Quadratic Y Y Y L qpsolve

SemiDef N Y Y L lmisolver

SemiDef Y N Y L semidef

N.L.S. N N N L optional Leastsq /

lsqrsolve

Non-linear Y N N L Y optim

Non-linear N N N s fminsearch

Objective Single

Objective

Continuous/

Differentiable

Fminsearch

Non Smooth optim

Discrete Optim_sa,

optim_ga

Optim_moga

Optim_nsga2

17www.esi-group.com

Linear problemmin𝑥

𝑓 𝑥 = 𝑐𝑇𝑥

Linear constraints

• Equalities𝐶1𝑇𝑥 = 𝑏1

• Inequalities𝐶2𝑇𝑥 ≥ 𝑏2

Linear optimization

18www.esi-group.com

Interior Points Method

« Affine Scaling »

Solve suite of optimization problems onellipsoid.

Ellipsoids size are decreasing at eachiteration by a given scale factor (between 0and 1).

Caracteristic:

• Polynomial time O(𝑛𝑚)

• Direct search

• (In)Equalities and bounds constraints

Linear optimization - karmarkar

19www.esi-group.com

Quadratic problem

minx𝑓 𝑥 =

2𝑥𝑇𝑄𝑥 + 𝑐𝑇𝑥

With Q la « hessian » and c « gradient »

of a virtual energy function

Linear constraints

• Equalities𝐶1𝑇𝑥 = 𝑏1

• Inequalities𝐶2𝑇𝑥 ≥ 𝑏2

Quadratic optimization

Concave shape, convex problem

20www.esi-group.com

Quadratic optimization - qpsolve

Goldfarb-Idnani solver

« Feasible active set method »

Find non-constrained global optimum,then add non-respected constraints oneafter the other (activation)

Caracteristics :

• Strictly convex problems

• (In)Equalities and bounds constraints

• Polynomial time O(𝑛𝑚)

• Direct search

21www.esi-group.com

Least squares problem

min𝑥

𝑓 𝑥 2

𝑓: 𝐼𝑅𝑛 → 𝐼𝑅𝑚

𝑛 number of unknonw, 𝑚 nb of points

𝑓 non necessarily linear or differentiable

Linear regression (QUADRATIC)

Minimize distance between measurements (𝑛 = 2) and an unknown line (𝑚 = 1)

2𝑄𝑥 − 𝑐 2 =

2(𝑥𝑇𝑄𝑇𝑄𝑥 − 2𝑥𝑇𝑄𝑇𝑐 + 𝑐𝑇𝑐)

Non Linear Least Squares

22www.esi-group.com

Non Linear Least Squares - lsqrsolve

Levenberg-Marquardt algorithm

To find a new direction:

• Gradient method (green) to provide steepestdescent direction. Efficient when far from theoptimal position.

• Gauss-Newton (red), approximate the problem aslocally quadratic and solve it to find new position.Efficient when close to the optimal position.

Caracteristics

• Polynomial time

23www.esi-group.com

Non Linear optimization - fminsearch

Nelder-Mead Simplex

Simplex manipulation

Caracteristics

• Non constrained

• High complexityBetter suited for small problem

• Direct search

3D Simplex

24www.esi-group.com

Quasi-Newton method

Quasi-Newton statement𝑥𝑘+1 − 𝑥𝑘 = 𝐵𝑘 𝑓 𝑥𝑘+1 − 𝑓 𝑥𝑘

⇒ 𝑥𝑘+1 ≈ 𝑥𝑘 − ρ𝑘𝐵𝑘𝑓(𝑥𝑘)

• Broyden-Fletcher-Goldfarbapproximation for pour Hessian 𝐵𝑘

• ρ𝑘 pour control convergence

Caracteristics

• High memory need to store 𝐵𝑘 : O(n²)(O(n) in case of limited method)

• Gradient info must be provided

• Polinomial time O(n²) but not accurate

Non Linear optimization - optim

• Fsolve (alternative)

• Datafit / leastsq (built upon optim)

25www.esi-group.com

GA Theory

- Natural selection -

Probabilistics and non deterministics transitons:

Selection, Cross-over et Mutation

Caracteristics

• Bounds constraints

• Non Linear, Non Convex

Remarks

optim_ga /optim_moga: Single/Multi objective

pareto_filter: Non-dominated sorting

Genetic Algorithms

26www.esi-group.com

Simulated Annealing

Simulated Annealing optim_sa

Empirical method based on metallurgicalprocess

Alternate slow cooling cycles with heatingcycles (annealing) in order to minimize materialinternal energy (strongest configuration)

Metropolis-Hastings Algorithm

Starting at a given state, we modify systemtowards another state. Either it get better(energy decrease) or it get worse.

Drawbacks:

• Empirical set-up

• Bounds constraints only

Benefits:

• Global optimum

• Discrete optimization

SCILAB WORKSHOP...approximation for pour Hessian 𝑘 • ρ𝑘 pour control convergence...

Documents

Hessian Clothes by Sonsvijay International Kolkata.ppsx

Pome 2002 ,Hessian,Maximum Entropy

The Geometry of Hessian Structures

2ª série€¦ · 10 = 6 = 4 =𝑘 , onde k é o fator de proporcionalidade. Assim teremos 10 =𝑘 → =10.𝑘 =𝑘 → =6.𝑘 4 = 𝑘→ 4. Como + + representa o prêmio recebido,

Binomial Coefficients Pascal’s Triangle€¦ · The core of this TRIANGLE is the Pascal’s Identity: Proof of Pascal’s identity: LHS: 𝑘 = 𝑛! 𝑘!𝑛−𝑘! RHS: −1

Statistical Shape Analysis - Scientific Computing and ... Shape Analysis It’s all about representation… 𝐩𝑘= 𝑘, 𝑘 T Configuration Space 𝑆𝑖= 𝐩𝑖 1 𝐩𝑖

MATTI LEHTONEN KYTKENTÄSUUNNITTELUN PROSESSI- JA ...¶.pdf · Kuluttajahintaindeksi vuonna 2005 𝑘 Kuluttajahintaindeksi vuonna 𝑘 𝑃 Pikajälleenkytkennöistä aiheutunut

An Investigation into Neural Net Optimization via Hessian ...proceedings.mlr.press/v97/ghorbani19b/ghorbani19b.pdf · Hessian Eigenvalue Densities for n > 107 To understand the Hessian,

Hessian Fly-Associated Bacteria: Transmission

Hessian Fly ID/Management in Wheat

QUANTITATIVE CARACTERISTICS FOR PORTS’ GENERATIONS: …

DC / AC Ratio Optimization · DC / AC Ratio Optimization 3 𝑟 𝑟𝑎 𝑖 = 𝑅𝑎 𝐴𝑟𝑟𝑎𝑦 𝑟 ( 𝑘 ) 𝑅𝑎 𝐼 𝑟 𝑟 (𝐴 𝑘 )

定性情報と企業価値...Fama-Macbeth regression : 𝑅𝑒 𝑖, = Ù+ 𝑘=1 5 Ú1,𝑘𝑇𝑤𝑖 𝑒 𝑖, −𝑘 𝐸 𝑒 𝑒+ 𝑘=1 5 Ú2,𝑘𝑇𝑤𝑖 𝑒 𝑖,

Lección 4myfaculty.metro.inter.edu/jahumada/mate1002/Unidad%204/...1=4, 2=7, 𝑘+2=3 𝑘+1−2 𝑘,para 𝑘≥1 • Solución: Deseamos determinar 3, 4, 5 𝑖 𝑘+2=3 𝑘+1−2

Clinical and Mechanical Caracteristics for Orthodontic Nitinol Wires · 2010. 12. 7. · Clinical and Mechanical Caracteristics for Orthodontic Nitinol Wires CAMELIA, SZUHANEK1, T.,

Hessian-based Analysis of Large Batch Training and ...papers.nips.cc/paper/7743-hessian-based-analysis... · with large Hessian spectrum show poor robustness to adversarial perturbation

SPECIFIC MANUAL TECHNICAL CARACTERISTICS MANUAL …

STATISTIKA DAN PROBABILITAS (CIV -110)...Contoh soal : Misal 5 buah koin ditos. Hitung peluang munculnya 2 gambar (2G) 𝑘 𝑁= ! 𝑘! −𝑘! = 5! 2!∗3! =10 KURVA NORMAL DISTRIBUSI

Biology and Management of Hessian Fly in the Southeast · into Long Island, New York, in straw bedding used by Hessian soldiers during the Revolutionary War. Hessian fly prefers to

Optimization Methods for Non-Linear/Non-Convex Learning … · Yann LeCun Making the Hessian better conditionedMaking the Hessian better conditioned The Hessian is the covariance