Upload
doananh
View
270
Download
0
Embed Size (px)
Citation preview
Numerical OptimizationIntroduction
Shirish Shevade
Computer Science and AutomationIndian Institute of ScienceBangalore 560 012, India.
NPTEL Course on Numerical Optimization
Shirish Shevade Numerical Optimization
Introduction
Optimization : The procedure or procedures used to make asystem or design as effective or functional as possible (adaptedfrom www.thefreedictionary.com)
Why Optimization?Helps improve the quality of decision-makingApplications in Engineering, Business, Economics,Science, Military Planning etc.
Shirish Shevade Numerical Optimization
Mathematical Program
Mathematical Program : A mathematical formulation of anoptimization problem:
Minimize f (x) subject to x ∈ S
Essential Components of a Mathematical program:
x : variables or parametersf : objective functionS: feasible region
What is a solution of this Mathematical Program?
x∗ ∈ S such that f (x∗) ≤ f (x) ∀ x ∈ Sx∗: solution, f (x∗): optimal objective function valuex∗ may not be unique and may not even exist.
Maximize f (x) ≡ − Minimize −f (x)
Shirish Shevade Numerical Optimization
Mathematical Optimization
The problem,
Minimize f (x) subject to x ∈ S
can be written as
minx
f (x)
s.t. x ∈ S (1)
Mathematical Optimization a.k.a. Mathematical programming
Study of problem formulations (1), existence of a solution,algorithms to seek a solution and analysis of solutions.
Shirish Shevade Numerical Optimization
Some Optimization Problems
Find the shortest path between the two points A and B in ahorizontal plane
Shirish Shevade Numerical Optimization
Some Optimization Problems
Bus Terminus Location Problem: Find the location of thebus terminus T on the road segment PQ such that thelengths of the roads linking T with the two cities A and B isminimum.
Shirish Shevade Numerical Optimization
Some Optimization Problems
Given two points A and B in a vertical plane, find a pathAPB which an object must follow, so that starting from A, itreaches B in the shortest time under its own gravity.
Shirish Shevade Numerical Optimization
Some Optimization Problems
Facility location problem: Find a location (within theboundary) that minimizes the sum of distances to each ofthe locations
Shirish Shevade Numerical Optimization
Some Optimization Problems
Transportation Problem: Find the “best" way to satisfy therequirement of demand points using the capacities ofsupply points.
Shirish Shevade Numerical Optimization
Some Optimization Problems
Data Fitting Problem: From a family of potential models,find a model that “best" fits the observed data.
−2 −1 0 1 2 3 4 5 60
2
4
6
8
10
12
14
16
18
20
Shirish Shevade Numerical Optimization
Some Optimization Problems
Application DomainsVarious disciplines in EngineeringScienceEconomics and StatisticsBusiness
Some Example ProblemsScheduling problemDiet ProblemPortfolio Allocation ProblemEngineering DesignManufacturingRobot Path Planning. . .
Shirish Shevade Numerical Optimization
Some Optimization Problems
Euclid’s Problem (4th century B.C.): In a given triangleABC, inscribe a parallelogram ADEF such that EF‖AB andDE‖AC and the area of this parallelogram is maximum.AM (Arithmetic Mean)-GM (Geometric Mean) Inequality:For any two non-negative numbers a and b,
√ab ≤ a + b
2
Problem: Find the maximum of the product of twonon-negative numbers whose sum is constant.Find the dimensions of the rectangular closed box ofcapacity V units which has the least surface area.
Shirish Shevade Numerical Optimization
Mathematical Optimization Process
Typical steps for Solving Mathematical Optimization Problems
Problem formulationChecking the existence of a solutionSolving the optimization problem, if a solution existsSolution analysisAlgorithm analysis
Shirish Shevade Numerical Optimization
Mathematical Optimization Process
Typical steps for Solving Mathematical Optimization Problems
Problem formulation
minx
f (x)
s.t. x ∈ S
Checking the existence of a solutionSolving the optimization problem, if a solution existsSolution analysisAlgorithm analysis
Shirish Shevade Numerical Optimization
Formulation: Bus Terminus Location Problem
Coordinates of A and B:xA = (xA1, xA2) andxB = (xB1, xB2)
Equation of line PQ:ax1 + bx2 + c = 0Use Euclidean distancexT = (xT1, xT2) (variables)The objective is to minimized(xA, xT ) + d(xB, xT )
T lies on PQ (constraint)
minxT1,xT2
d(xA, xT ) + d(xB, xT )
s.t. axT1 + bxT2 + c = 0
Shirish Shevade Numerical Optimization
Formulation: Facility Location Problem
xA, xB, xC and xD belong to therespective location boundariesUse Euclidean distance(xP1, xP2) (variables)The objective is to minimized(xA, xP) + d(xB, xP) +d(xC , xP) + d(xD, xP)
xA ∈ A, xB ∈ B, xC ∈ C andxD ∈ D (constraints)
minxP1,xP2
d(xA, xP) + d(xB, xP) + d(xC , xP) + d(xD, xP)
s.t. xA ∈ A, xB ∈ B, xC ∈ C, xD ∈ D
Shirish Shevade Numerical Optimization
Formulation: Transportation Problem
minxij
i=1,2;j=1,2,3
∑ij cijxij
s.t.∑3
j=1 xij ≤ ai , i = 1, 2∑2i=1 xij ≥ bj , j = 1, 2, 3
xij ≥ 0 ∀ i , j
ai : Capacity of the plant Fibj : Demand of the outlet Rjcij : Cost of shipping one unitof product from Fi to Rjxij : Number of units of theproduct shipped from Fi to Rj(variables)The objective is to minimize∑
ij cijxij∑3j=1 xij ≤ ai , i = 1, 2
(constraints)∑2i=1 xij ≥ bj , j = 1, 2, 3
(constraints)xij ≥ 0 ∀ i , j (constraints)
Shirish Shevade Numerical Optimization
Formulation: Data Fitting Problem
−2 −1 0 1 2 3 4 5 60
2
4
6
8
10
12
14
16
18
20 Given : {xi , yi}ni=1, n data
pointsGiven : Most probable modeltype, f (x) = ax2 + bx + ca, b, c: variablesMeasure of misfit: (y − f (x))2
The objective is to minimize∑i (yi − (ax2
i + bxi + c))2
No constraints
mina,b,c
∑ni=1 (yi − (ax2
i + bxi + c))2
Shirish Shevade Numerical Optimization
Mathematical Optimization Process
Typical steps for Solving Mathematical Optimization Problems
Problem formulationChecking the existence of a solutionSolving the optimization problem, if a solution exists
Graphical methodAnalytical methodNumerical method
Solution analysisAlgorithm analysis
Shirish Shevade Numerical Optimization
Functions of One Variable
f (x) = (x − 2)2 + 3Minimum at x∗ = 2, minimum function value: f (x∗) = 3
−2 −1 0 1 2 3 4 5 60
4
8
12
16
20
x
f(x)
Shirish Shevade Numerical Optimization
Functions of One Variable
f (x) = −x2
Maximum at x∗ = 0, maximum function value: f (x∗) = 0
−4 −3 −2 −1 0 1 2 3 4−16
−14
−12
−10
−8
−6
−4
−2
0
2
x
f(x)
Shirish Shevade Numerical Optimization
Functions of One Variable
f (x) = x3
Saddle Point at x∗ = 0
−20 −15 −10 −5 0 5 10 15 20−8000
−6000
−4000
−2000
0
2000
4000
6000
8000
x
f(x)
Shirish Shevade Numerical Optimization
Functions of One Variable
A typical nonlinear function
Shirish Shevade Numerical Optimization
Functions of Two Variables: Surface Plots
f (x1, x2) = x21 + x2
2Minimum at x∗
1 = 0, x∗2 = 0; f (x∗
1 , x∗2 ) = 0
−3−2
−10
12
3
−3
−2
−1
0
1
2
30
5
10
15
20
x1x2
f(x1
,x2)
Shirish Shevade Numerical Optimization
Functions of Two Variables: Contour Plots
f (x1, x2) = x21 + x2
2
−3 −2 −1 0 1 2 3−3
−2
−1
0
1
2
3
x1
x2
1
1
1
2
2
2
2
2
4
4
4
4
4
46
6
6
6
6
66
6
8
8
8
8
8
8
88
8
0.1
Shirish Shevade Numerical Optimization
Functions of Two Variables: Surface Plots
f (x1, x2) = x1 exp(−x21 − x2
2 )
Minimum at (−1/√
2, 0), maximum at (1/√
2, 0)
−2
−1
0
1
2
−2
−1
0
1
2−0.5
0
0.5
x1x2
f(x1
,x2)
Shirish Shevade Numerical Optimization
Functions of Two Variables: Contour Plots
f (x1, x2) = x1 exp(−x21 − x2
2 )
−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2
−1.5
−1
−0.5
0
0.5
1
1.5
2
2.5
3
x1
x2
−0.4
−0.3
−0.3
−0.2
−0.2
−0.2
−0.1
−0.1
−0.1
−0.1
0.1
0.10.
1
0.1
0.2
0.2
0.2
0.3
0.30.4
Shirish Shevade Numerical Optimization
Functions of Two Variables: Contour Plots
f (x1, x2) = f (x1, x2) = 4x21 + x2
2 − 2x1x2
−3 −2 −1 0 1 2 3−3
−2
−1
0
1
2
3
x1
x2
0.1
0.5
0.5
1
1
1
22
2
2
4
4
4
4
4
8
8
8
8
8
8
Shirish Shevade Numerical Optimization
Functions of Two Variables: Contour Plots
Rosenbrock function: f (x1, x2) = 100(x2 − x21 )
2+ (1− x1)
2
Minimum at (1, 1)
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
x1
x2
1
1
1
1
1 1
1
2
2
2
2
2
4
44
4
4 4
4
4
4
8
8
88
8
8
8
8
8
816
1616
16
16
16
16 16
16
16
Shirish Shevade Numerical Optimization
Functions of Two Variables: Surface Plots
010
2030
4050
0
10
20
30
40
50−10
−5
0
5
10
x1x2
f(x1,x
2)
Shirish Shevade Numerical Optimization
Mathematical Optimization Process
Typical steps for Solving Mathematical Optimization Problems
Problem formulationChecking the existence of a solutionSolving the optimization problem, if a solution exists
Graphical methodAnalytical methodNumerical method
Solution analysisAlgorithm analysis
Shirish Shevade Numerical Optimization
Iterates of an Optimization Algorithm
f (x1, x2) = (x1 − 7)2 + (x2 − 2)2
Initial Point: (6.4, 1.2), Minimum at (7, 2)
6 6.2 6.4 6.6 6.8 7 7.2 7.4 7.6 7.8 81
1.2
1.4
1.6
1.8
2
2.2
2.4
2.6
2.8
3
X1
X2
0.1
0.1
0.1
0.5
0.5
0.5
0.5
0.5
0.5
1
1
11
1
1
1
1
1
Shirish Shevade Numerical Optimization
Iterates of an Optimization Algorithm
f (x1, x2) = 4x21 + x2
2 − 2x1x2Initial Point: (−1,−2), Minimum at (0, 0)
−3 −2 −1 0 1 2 3−3
−2
−1
0
1
2
3
X1
X2
0.1
0.50.
5
1
1
1
22
2
2
4
4
4
4
4
8
8
8
88
8
Shirish Shevade Numerical Optimization
Iterates of an Optimization Algorithm
f (x1, x2) = 100(x2 − x21 )
2+ (1− x1)
2
Initial Point: (0.6, 0.6), Minimum at (1, 1)
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
x1
x2
1
1
1
1
1
12
2
2
2
2
4
4
44
4
4
4
4
8
8
8
8
8
8
8
8
8
16
16
16
16
16
16
16
16
Shirish Shevade Numerical Optimization
Iterates of an Optimization Algorithm
f (x1, x2) = 100(x2 − x21 )
2+ (1− x1)
2
Initial Point: (−2, 1), Minimum at (1, 1)
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
x1
x2
1
11
1
1
12
2
2
2
2
4
4
4
4
4
4
4
4
8
8
8
8
8
8
8
8
8
16
16
16
16
16
16
16
16
Shirish Shevade Numerical Optimization
A solution to Data Fitting Problem
−2 −1 0 1 2 3 4 5 60
2
4
6
8
10
12
14
16
18
20
Shirish Shevade Numerical Optimization
Types of Optimization Problems
Constrained and unconstrained optimizationContinuous and discrete optimizationStochastic and deterministic optimization
Shirish Shevade Numerical Optimization
Types of Optimization Problems
Constrained optimization problem:
minx
f (x)
s.t. x ∈ S
Unconstrained optimization problem:
minx
f (x)
Shirish Shevade Numerical Optimization
Types of Optimization Problems
Continuous optimizationVariables are typically real-valued
Discrete optimizationVariables are not real-valued: they take binary or integervalues
Shirish Shevade Numerical Optimization
Types of Optimization Problems
Stochastic optimizationSome or all of the problem data are randomIn some cases, the constraints hold with some probabilitiesNeed to define feasibility and optimality appropriately
Deterministic optimizationNo randomness in problem data and constraints
Shirish Shevade Numerical Optimization
Types of Optimization Problems
Constrained and unconstrained optimizationContinuous and discrete optimizationStochastic and deterministic optimization
Shirish Shevade Numerical Optimization
Types of Optimization Algorithms
Local optimization algorithmsFind “locally" optimal solutions
Global optimization algorithmsFind the “best" solution among all locally optimal solutions
Shirish Shevade Numerical Optimization
Overview
Mathematical BackgroundOne dimensional unconstrained optimization problemsAlgorithms for multi-dimensional unconstrainedoptimization problemsMulti-dimensional Constrained optimization problemsActive Set MethodsPenalty and Barrier Function Methods
Shirish Shevade Numerical Optimization
Some References
Fletcher R., Practical Methods of Optimization, Wiley(2000)Kambo N. S., Mathematical Programming Techniques,Affiliated East-West Press (1984)Luenberger D., Linear and Nonlinear Programming,Addison-Wesley (1984)Nocedal J. and Wright M., Numerical Optimization,Springer (2000)Suresh Chandra, Jayadeva and Mehra A., NumericalOptimization with Applications, Narosa (2009)
Additional Reading List:Tikhomirov V. M., Stories About Maxima and Minima, TheAmerican Mathematical Society (1990), Universities Press(India) Limited (1998).
Shirish Shevade Numerical Optimization