AIAA_RDO

American Institute of Aeronautics and Astronautics

1

Comparison of Different Uses of Metamodels for Robust Design Optimization

Johan A. Persson1 and Johan lvander2 Linkping University, 581 83, Linkping, Sweden

This paper compares different approaches for using kriging metamodels for robust design optimization, with the aim of improving the knowledge of the performance of the approaches. A popular approach is to first fit a metamodel to the original model and then perform the robust design optimization on the metamodel. However, it is also possible to create metamodels during the optimization. Additionally, the metamodel need not necessarily reanimate the original model; it may also model the mean value, variance or the actual objective function. The comparisons are based on two analytical functions and a dynamic simulation model of an aircraft system as an engineering application. In the comparisons, it is seen that creating a global metamodel during the optimization slightly outperforms the other approaches that involve metamodels.

Nomenclature A2 = an area of the piston found inside the main valve in the dynamic pressure regulator As = an area of the piston found inside the main valve in the dynamic pressure regulator i = the ith regression coefficient k = number of points used for the complex algorithm = the mean value mp = the mass of the piston inside the main valve n = number of optimizations ncalls = required number of function evaluations needed for the optimization algorithm to converge P = the probability of finding the global optimum by performing one optimization R(x1,x2) = the correlation function between x1 and x2 s

= an estimation of the standard deviation

2

= the variance tfill = the time it takes to fill the environmental system xi = value of the ith variable Z(x) = A stochastic process

I. Introduction obust design optimization (RDO) may be used in the design process to find optimal designs which are insensitive to uncertainties and errors.1-3 Since decisions about designs need to be taken by the developers even

though information is lacking, there will always be uncertainties and variations present.4,5 These may among other things stem from measurements, manufacturing tolerances and environmental conditions.6

Optimization Algorithm

Objective Function

Estimate Statistics

Model

Figure 1. A general workflow of a Robust Design Optimization process.

1 PhD Student, Department of Management and Engineering, [email protected].

2 Professor, Department of Management and Engineering, [email protected].

R


2

A general workflow of an RDO is shown in Figure 1. The mean value and standard deviation are estimated each time the objective function value is called by the optimization algorithm. Consequently RDO requires numerous simulations of the computer model. To avoid unrealistically long simulation times and occupation of precious resources, methods have been suggested where computationally effective global metamodels are used instead of the original models.1 A metamodel is a numerical model of how a quantity varies when the parameters that are affecting it are varied and consequently it is possible to replace each of the activities in Figure 1 with a metamodel. Several possible RDO approaches therefore exist, where different entities are replaced by metamodels.

A common approach is to fit a global metamodel to the desired output of the original model.7-9 The global model is then called to estimate the objective function in the RDO. Since the position of the optimum is to some degree unknown beforehand, the metamodel needs to reanimate the whole design space accurately. This requires numerous simulations of the original model and some of them may be in areas which the optimization algorithm never reaches. Another approach is to fit a global metamodel to the original model as the RDO progresses, meaning that the only simulations of the original model that are performed are made with parameter values which are used by the RDO.10

The aim of this paper is to compare several RDO approaches, where different entities are replaced by metamodels, for two analytical examples and an aircraft system model. Apart from giving insights about the modeled system, a comparison increases knowledge of when to apply the different methods. There exist numerous types of metamodels and to narrow down the number of compared approaches in this paper, only one type of metamodel is used. While many comparisons of metamodels have been made,11-14 none stand out as clearly superior9. However, the kriging metamodel performs well in the comparisons and is also applied to RDO problems by several authors.15-17 It has therefore been chosen as metamodel for this comparison.

To further decrease the number of model calls, an efficient sampling method and optimization algorithm are chosen. Latin Hypercube Sampling (LHS) is an efficient sampling based method which has been proven to yield accurate estimations for just tens of samples,18,19 and is therefore chosen as the sampling method in this work. The Complex-RF algorithm is a fairly efficient optimization algorithm and its performance has been demonstrated for system models.20,21 The same sampling methods and optimization algorithms are used in all approaches to make the comparison fair and keep differences to a minimum.

A brief description of Robust Design Optimization can be found in section II, together with descriptions of Latin Hypercube Sampling, Kriging and Complex-RF. The compared approaches are presented in section III and the comparison is presented in section IV. The final sections, V and VI, respectively comprise a discussion and the conclusions.

II. Robust Design Optimization It is possible to find an optimal solution which is insensitive to uncertainties and errors by performing an RDO. This is achieved by optimizing the mean value and standard deviation of the desired system properties. Mean values and standard deviations are statistical entities which can be estimated in different ways.

For all methods, the model parameters which should be used as variables in the robust design optimization are identified and their uncertainties estimated or measured. The parameters which influence the system properties of interest may be identified by performing a local sensitivity analysis22,23 or a screening24,25. By assigning the values of the model parameters to standard probability distributions, it is possible to describe the characteristics of the uncertainties with just a few parameters.

Many methods exist for estimating how the uncertainties and errors in the model parameters affect the system properties of interest and a few are presented in Ref 7 and Ref 19. Generally, the methods return probability distributions for the system properties of interest. Those may then be used for taking decisions based on probabilities. A combination of the mean values and standard deviations of those distributions are used as an objective function for RDO, while the probability of failure is used as the objective function for Reliability-Based Design Optimization (RBDO).1

In this paper, sampling based approaches are used to estimate the probability distributions of the system characteristics. The most renowned sampling method is probably the Monte Carlo Simulation (MCS) which draws random samples from the probability distributions of the input parameters. As the number of samples increases, the estimation will converge towards the real probability distributions according to the law of large numbers, but this might mean tens of thousands of samples.19 This is unrealistic for RDO since the probability distributions need to be calculated each time the objective function is called. A more effective sampling method, Latin Hypercube Sampling, is instead used for the sampling in this paper.


3

A. Latin Hypercube Sampling LHS is an effective sampling method where the design space is divided into n intervals of equal probability.18

One sample is drawn from each interval which ensures that the n samples are spread around the design space. It has been shown to give accurate estimations of the mean value and standard deviation of the model output for less than one hundred samples.18,19 One advantage of LHS is that the required number of samples does not scale with the number of design parameters. A drawback compared to MCS, however, is that it is difficult to add samples to a performed LHS. For MCS it is possible to add samples until the desired error is reached, whereas it is only possible to increase the LHS by doubling the number of samples.

B. Kriging Originally, kriging metamodels were used for geostatistical applications but they have also been demonstrated to

perform well for high fidelity computer models.11,15 A kriging model is a group of interpolation techniques for spatially correlated data and is based on a stochastic model. When a kriging model is used as a deterministic metamodel, the expected value at the desired point is used as the approximate value at the point.16

A kriging model is a combination of a polynomial model and local deviations of the form seen in Eq. (1).

( ) ( )=

+=k

iii xZxfy

1

(1)

i is the ith regression coefficient, whereas fi(x) is a function of the variables x and Z(x) a stochastic process with zero mean and a spatial correlation function given by Eq. (2).

( ) ( )[ ] ( )jiji xxRxZxZCov ,, 2= (2)

2 is the variance of the process and R(xi,xj) is a correlation function between xi and xj. The first term in Eq. (1)

is an approximation of the trend of the model output, resembling a polynomial metamodel, and is modified according to the problem.17 The correlation function includes unknown parameters and unfortunately it is time-consuming to determine the values of the parameters. This is an optimization problem in its own right and may be solved by using maximum likelihood estimation; see for example.17

C. Complex-RF The Complex-RF optimization algorithm was developed from the Nelder-Mead Simplex method. The algorithm

uses k number of points for converging and in each iteration the worst point is moved along a line through the center of the other points until it is no longer the worst.20,21,26 Another point is now the worst and consequently moved until another point is worse. This process continues until the stop criterion is met or the maximum number of evaluations is reached.

Since the Simplex and Complex methods use several starting points and spread them around the design space, they are suitable for problems which are sensitive to the choice of starting point. Furthermore, they do not use any derivatives of the objective function and are not dependent on the linearity or differentiability of the objective functions, and hence are applicable to a wide range of problems.

III. Compared Approaches This section presents six different approaches for performing robust design optimization which are compared in

this paper. Schematic workflows for the six approaches can be seen in Figure 2.

A. Approach 1 The first approach is the most straightforward and almost a brute force method. Each time the optimization

algorithm requests the value of the objective function, the mean value and standard deviation of the model parameters are sent to the Latin Hypercube Sampler. The sampler draws samples from the true model and estimates the mean value and standard deviation of the system properties of interest. A combination of these values is used to calculate the objective function. This process can be seen in Figure 2a).


4

B. Approach 2 Shown in Figure 2b), this approach starts by drawing a predefined number of samples from the original model by

means of LHS, which is used to spread samples for metamodels by several authors.16,17 These samples are used to fit a global kriging model to the output of the original model. The optimization then works as in Approach 1, with one exception. Each time LHS draws samples, they are drawn from the kriging model instead of the original model. Consequently, the only simulations of the original model are the ones that are performed to fit the kriging model. This is probably the most widely used approach where metamodels are utilized.

C. Approach 3 The workflow can be seen in Figure 2c) and at the beginning of the optimization process this approach performs

the same operations as Approach 1. However, all calls of the original model by the LHS are stored until a predefined number of such calls have been made. A kriging model of the output of the original model is now fitted by using these model calls as samples. During the rest of the optimization, the LHS calls the kriging model instead of the

Set up

problem

Create a MM of the

system

Draw samples from

the model to

estimate mean and

standard deviation

Convergence?

Optimal

solution

Build objective

function from mean

and standard

deviation

Yes

No

Optimization

Does MM

exist?

No

Enough

samples to fit

a MM?

Yes

No

Draw samples from

the MM to estimate

mean and standard

deviation

Build objective

function from mean

and standard

deviation

Convergence?

Yes

No

Yes

Ne

w v

ari

ab

le v

alu

es

Ne

w v

aria

ble

va

lue

s

a) Approach 1 b) Approach 2 c) Approach 3

d) Approach 4 e) Approach 5 Figure 2. Workflows for the five approaches


5

original model to draw the remaining samples. As in Approach 2, the samples required to fit the kriging model are the only simulations of the original model that are performed.

D. Approach 4 In this approach, shown in Figure 2d), the optimization process progresses as in Approach 1 until a predefined

number of mean values and standard deviations have been estimated. These estimations are used as samples to fit one kriging model of how the mean value varies in the design space, and one kriging model for the variation of the standard deviation. When the kriging models have been created, the objective function calls the two metamodels in order to calculate its value during the rest of the optimization. The approach with creating metamodels of the mean value and standard deviation instead of the original model itself has been demonstrated by for example Lnn et al.27

This method has a similar drawback to Approach 1. The LHS draws samples from the original model each time the mean value and standard deviation are estimated until they have been estimated the required number of times to enable the fitting of kriging models. Consequently, the required number of simulations of the real model is the number of samples drawn to estimate each mean value multiplied by the required number of mean values to fit a kriging model.

E. Approach 5 As can be seen from Figure 2e), this approach is rather similar to Approach 4, but instead of creating kriging

models of the mean value and standard deviation, a kriging model is fitted directly to the objective function value. When the predefined number of objective function values have been estimated using LHS, a kriging model is fitted. During the remainder of the optimization, the estimations of the objective function are calculated from the kriging model.

F. Approach 6 The comparison that is seen in section IV reveals that approaches 3 to 5 are inferior to approach 2, with approach

3 superior to 4 and 5. This would lead to the assumption that it is inferior to create a metamodel during the optimization compared to creating it before the optimization is started. However, it should be possible to find an

Ne

w v

aria

ble

va

lues

Figure 3. Workflow for the sixth approach.


6

approach that fits a metamodel during the optimization, which performs better than approach 2. Therefore, a sixth approach with the workflow seen in Figure 3 is suggested. The comparison in section IV also suggests that it is better to let the metamodel reanimate the response from the original model than the mean value and standard deviation or the whole objective function. The metamodels in Approach 6 therefore reanimate the response of the original model.

Approach 6 starts by spreading k points in the design space similar to the start of the Complex optimization algorithm. The deterministic values at the points are calculated by calling the original function. A metamodel is then fitted to the values. LHS is then used to estimate the mean values and standard deviations at the k points by calling the metamodel. The number of points, k, is set to eight for the comparison in this paper. k needs to be large enough to enable a metamodel to be fitted in the initial stage, but it is also desirable to keep it small in order to save as many calls to the original model as possible for later iterations of the optimization process.

As long as the maximum number of calls to the original model is not, the deterministic value at the point is calculated by calling the original model reached each time the optimization algorithm wants to estimate the objective function value. The surrogate model is then updated by including the new value in the samples that are used to fit the surrogate model.

Latin Hypercube Sampling calls the surrogate model to estimate the mean value and standard deviation that are needed to determine the value of the objective function. This means that only one call to the original model is made each time the value of the objective function is calculated, until the maximum number of calls to the original function is reached.

IV. Comparison of the Approaches The performance of each of the presented approaches is compared for two analytical functions and a dynamic

system model of an airplane system. For an optimization algorithm, the performance of interest is the chance of finding the global optimum, denoted hit-rate, and the number of function calls made, ncalls. To estimate the hit-rate, 1,000 optimizations are made for each approach and problem.

For RDO, it is desirable to find a design where the expected performance is as optimal as possible, while the dispersion is minimal. This can be formulated as an objective function with a linear combination of the mean value, , and standard deviation, s, of the stochastic model output. Since the mean value and standard deviation of a stochastic response often are unknown, they need to be approximated. One option is to estimate them according to Eq. (3) by drawing samples using LHS. For the objective functions in this paper, the weight of the mean value is set to one and the weight of the standard deviation is set to three, as shown in Eq. (3). The higher weight for the standard deviation specifies that in this case a design with a small dispersion in performance is more important than one with a better mean value.

( ) ( )( ) ( )( )xgxgxf 3min +=

( )( ) ( )=

=

m

jjxg

mxg

1

1

( )( ) ( )( )=

=

m

jjxg

mxg

1

2

11

(3)

A. Performance Index It is desirable to enable comparison of the performance of different optimization algorithms by using a numerical

value. A numerical value may also be used as an objective function for optimizing the parameters of optimization algorithms. The performance index suggested in this paper is derived as follows.

If P is the chance of finding the global optimum by performing one optimization, the hit-rate, then Eq. (4) specifies the probability of not finding the optimum by performing n optimizations.

( )nfoundnot PP = 1_ (4)


7

Consequently, Eq. (5) describes the probability of finding the optimum by performing n optimizations.

( )nfoundnotfound PPP == 111 _ (5)

To make a fair comparison between methods which require different numbers of function calls, n should be replaced by the allowed number of function calls divided by the required number of function calls, ncalls, as shown in Eq. (6). The allowed number of function calls depends on the problem, but as long as it is the same for all compared methods, the actual number is not important. Here, the allowed number of function calls is set to 100, meaning that the performance index is the probability of finding the global optimum if 100 calls of the original model are made.

( ) callsnPindexPerf /10011_ = (6)

This performance index is not adequate for trivial problems where the hit-rate is 100%, since Eq. (6) will equal one if P = 1 regardless of the required number of model calls. But if two optimization algorithms have the same hit-rate, it is just a matter of comparing the required number of model calls for the two algorithms, since the one requiring the fewest number of model calls is the better.

B. The Peaks function The approaches are compared for the Peaks function which is implemented in MATLAB and is called by writing

peaks in the command window. Here, it is shown in Figure 4 and Eq. (7).

( ) ( ) ( ){ } { } ( ){ }22212221523112221211 1exp31exp5101exp13 xxxxxxxxxxg + +=x subject to

2,1,33 = ixi (7)

It is a multimodal function, with a global deterministic minimum in [0.231 -1.626], whereas [-1.348 0.205] is a local deterministic minimum. For deterministic optimizations with Complex-RF, the probability of finding the global optimum is 73%.20 To convert the deterministic problem into a robustness problem, randomness is introduced by allowing the parameter values of a point to vary in an interval of 0.1 around the point.

The optimal point from a robust point of view depends on how much the standard deviation is weighted in relation to the mean value. If Figure 4 had been a plot of how the performance of a product varies with varying modeling parameters, it would probably be desirable to choose the global deterministic optimum as the design that should be realized. Even though the slopes are steep around the global deterministic optimum, its performance is

Figure 4. Graph over how the value of the Peaks function varies in the design space.


8

still better than most other design points. Consequently, the difference with a deterministic optimization is that it is computationally more expensive to perform a robust design optimization since the mean value and standard deviation need to be estimated for each design point. The criterion for a successful optimization is chosen to be an optimization where the optimal solution lies in the vicinity of the global deterministic optimum, i.e. fulfills 0.2 < x1 < 0.3, -1.7 < x2 < -1.6.

The comparisons of the different approaches can be seen in Table 1. The aim is to make the comparisons fair by allowing the same number of calls of the Peaks function for all approaches where a predefined number of calls can be set.

The row named hit-rate indicates what percentage of the 1,000 optimizations performed converged to a design point fulfilling the criterion for a successful optimization. A high number is advantageous since it indicates that the probability of finding the global optimum by performing one optimization is high.

The number of objective function evaluations is a measurement of how many times the optimization algorithm calculated the value of the objective function, either by calling a kriging metamodel (A4 & A5) or performing an LHS of a kriging model (A2, A3 and A6) or the original model (A1). Since it can be seen as a measure of how many operations the algorithm performs, it is an advantage to have as few objective function evaluations as possible.

The number of samples for each LHS indicates how many samples of the original model or kriging model each time an LHS is performed. A high number is advantageous since having more samples improves the accuracy of the estimations of the mean value and standard deviations. However, a high number also means that more samples need to be drawn each time an LHS is performed and consequently the computational time will increase.

The next row indicates the chosen number of samples used to fit the kriging models for each approach. No kriging model is created for Approach 1, whereas Approach 2, 3 and 6 fit their kriging models from 100 samples of the original model in Table 1. From the same table it can be seen that the kriging models in Approach 4 are fitted when 10 mean values and standard deviations have been estimated by performing 10 LHSs of the original model. Similarly, a kriging model is created in Approach 5 when 10 objective function values have been estimated.

The number of calls of the original model is an interesting entity since it is used to estimate the performance index of the optimization algorithms. For expensive models, each simulation of the original model might be more computationally demanding than all the operations performed by the optimization algorithm. Consequently, the number of calls of the original model determines the wall clock time for the robust design optimization process for expensive models.

In the bottommost row, the performance index for each optimization can be seen. This is the most revealing characteristics of the performance of each approach for the Peaks function, since it is an estimation of how probable it is that the global optimum will be found by performing 100 simulations of the original model. As mentioned in Eq. (6), it is calculated from the hit-rate and the required number of calls of the original model, which corresponds to rows three and seven in the table.

Approach 6 is superior to the other approaches according to the performance indices in Table 1. The large number of objective function evaluations for Approach 6 is not a great issue since most of them are made on the computationally effective metamodel.

The large number of calls of the original model for Approach 1 deteriorates its performance index making it a poor choice even though its hit-rate is the second highest. Approaches 2 and 3 have significantly higher hit-rates than Approach 4 and Approach 5, where Approach 2 is the best and Approach 5 the worst. The hit-rates in Approach 4 and Approach 5 are close to zero, indicating that the probability of finding the global optimum is minimal, making them unsuitable for solving this problem.

Table 1. Comparison of the five approaches for 100 allowed calls of Peaks. Approach A1 A2 A3 A4 A5 A6 Hit-Rate [%] 59.0 42.8 30.9 0.3 0.1 86.0 Number of objective function evaluations 628 602 544 95 94 2000 Number of samples for LHS 10 20 10 10 10 10 Number of samples for kriging - 100 100 10 10 100 Number of calls of the original model 6280 100 100 100 100 100 Performance Index 0.014 0.428 0.309 0.003 0.001 0.86


9

C. The Aspenberg Function An interesting function from a robustness point of view is the function presented for one variable in Ref 28.

Figure 5a) shows how its function value changes when the value of its variable is altered. For this comparison it has been modified to be a function of two variables, shown in Eq. (8), to increase the complexity of the function.

( ) ( )++= n

ii

n

ii xxn

ii eexxg

22 25.010302

2 4

subject to

2,1,33 = ixi (8)

The function has a deterministic optimum in [-0.280 -0.280], which is almost always found for deterministic optimizations using Complex-RF. For the objective function that is presented in Eq. (3) it has robust optimums in [-0.25 0.10] and [0.10 -0.25], which can be seen in Figure 5b). The criterion for a successful optimization is chosen to be when the optimization algorithm suggests a point fulfilling either -0.3 < x1 < -0.2, 0.01< x2 < 0.18 or 0.01 < x1 < 0.18, -0.3< x2 < -0.2. These two parts of the design space are similar and therefore it does not matter which of them the suggested optimum belongs to.

In Table 2, a comparison of the different approaches can be seen for 100 samples. It is worth noting that Approach 1 finds the robust optimum almost every time. Naturally, this leads to a reasonably high performance index even though the number of calls of the function is huge. 1310 simulations of a computationally expensive model are probably unrealistically many. Approach 2 and Approach 3 need over a thousand objective function evaluations to converge, but as most of the evaluations are made on a computationally cheap kriging model it is not an important issue. Approaches 4 and 5 display poor hit-rates for this problem and consequently get worse performance indices than the other approaches. The sixth approach gets the highest performance index, mainly due to its high hit-rate.

-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10

0.5

1

1.5

2

2.5

3

3.5

4

4.5The Aspenberg function for one variable

x1

g(x)

x1

x2

The value of Eq. 3 for the Aspenberg function

-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

a) 1D b) 3D-figure Figure 5. Two Graphs that shows a) the deterministic value of Eq. (8) for one variable and b) the value of

Eq. (3) for two variables.


10

D. An Aircraft System A Dynamic Pressure Regulator The aircraft system used for this comparison is the design of a dynamic pressure regulator (DPR) used to control

the air pressure delivered to an environmental control system. The system is modeled in Dymola and two screenshots from Dymola can be seen in Figure 6. A more thorough description can be found in.19

The most important function of the DPR is to ensure that the control system is filled as fast as possible. Consequently, the filling time, tfill, is the most important system characteristics. It is desirable to find the parameters that effect the characteristics most to improve the performance of the DPR. One way to identify important parameters is to perform a local sensitivity analysis in a reference design point.22 Partial derivatives for how each parameter affects the system characteristics are received by varying one parameter at a time. The normalized sensitivities from a local sensitivity are shown in Table 3. It can be seen that the three most important parameters are the areas of the piston inside the main valve, A2 and As, and the mass of the piston, mp.

Table 2. Comparison of the five approaches for 100 allowed calls of Aspenbergs function. Approach A1 A2 A3 A4 A5 A6

Hit-Rate [%] 99.9 24.6 21.2 5.3 6.2 99.0 Number of objective function evaluations 131 1200 1189 137 140 2000

Number of samples for LHS 10 10 10 10 10 10 Number of samples for kriging - 100 100 10 10 100

Number of calls of the original model 1310 100 100 100 100 100 Performance Index 0.410 0.227 0.212 0.053 0.062 0.99

Figure 6. Screenshots from Dymola that display the pressure regulator and the components inside it.

Table 3. Normalized sensitivities for a specific design of the dynamic pressure regulator

Tank

pres

sure

, p t

Max

o

penin

g ar

ea of

on/o

ff-val

ve

Max

o

penin

g ar

ea of

ven

t val

ve,

A

v

Max

o

penin

g ar

ea of

mai

n val

ve,

A

m

Fric

tion co

effic

ient

in m

ain val

ve

Mas

s of p

isto

n, m

p

Prel

oad

ing

of p

isto

n

Pist

on ar

ea fo

r

outp

ut f

low

, A

2

Pist

on ar

ea fo

r

supp

ort

pr

essu

re A

s

tf 1.4E-4 -5.7E-5 -8.4E-10 2.5E-4 0.0075 0.50 4.6E-4 -0.25 -0.26


11

When the important system characteristics and parameters are identified, it is time to set up an objective function. It is desirable to minimize both the time it takes to fill the control system and its standard deviation. The mathematical formulation can be found in Eq. (9).

( ) ( )( ) ( )( )xtxtxf fillfill 3min +=

( )( ) ( )=

=

m

jjxg

mxg

1

1

( )( ) ( )( )=

=

m

jjxg

mxg

1

2

11

subject to

61

4 1010 x

52

6 1010 x

0.11.0 3 x (9)

The system parameters are considered to follow uniform distributions with a spread of 10% around their mean values. This leads to Table 4, which contains statistics for the different approaches. Fortunately, all the approaches which involve metamodels perform better than the brute force method even for this engineering application. Approach 6 performs best, followed by Approach 2 and 3.

V. Discussion Approach 1, which does not involve any metamodels, has the highest hit-rate. However, thousands of calls of the

original model are needed for the algorithm to converge. This number of simulations is probably unrealistically large for computationally expensive models. If it is realistic to perform so many model simulations it is doubtful whether the model should be described as computationally expensive. The fact that the approaches that involve metamodels need more objective function evaluations than Approach 1 is not a major issue, since most are computationally cheap calls of a kriging metamodel.

The common approach of fitting a global metamodel to the output of the original model and performing the optimization on the metamodel has its advantages. The global metamodel may be used instead of the original model for other analyses, which can reduce the time for analyses significantly. In the other approaches that involve metamodels, the metamodels are fitted with the aim of improving accuracy in the vicinity of the optimum of the performed optimization and consequently lose precision in other parts of the design space, which might be interesting for other analyses. Optimization algorithms with high hit-rates, but requires many function evaluations to

Table 4. Comparison of the five approaches for 100 allowed calls of the dynamic pressure regulator.

Approach A1 A2 A3 A4 A5 A6

Hit-Rate [%] 84.0 51.3 47.0 26.0 22.8 76.0 Number of objective function evaluations 499 4000 4000 4000 4000 2000

Number of samples for LHS 5 5 5 5 5 5 Number of samples for kriging - 50 50 10 10 50

Number of calls of the original model 2495 50 50 50 50 50 Performance Index 0.070 0.762 0.719 0.452 0.404 0.942


12

converge, may be used to perform the optimization since all function evaluations are made by calling the metamodel.

It might also be that the original model is too computationally expensive to enable tens of simulations to be run, making Approach 2 the only realistic approach. Naturally, its precision decreases with a decreasing number of samples, but so does the precision of the others. Approaches 4 and 5 require several simulations of the original model for each sample that can be used to fit the metamodels. If few calls of the original model are allowed, the metamodels might need to be fitted using only a few samples which leads to low accuracy.

The poor performance of Approach 3 compared to Approach 2 can be explained by the organization of the samples used for creating the metamodel. Since the LHS draws samples to estimate the mean value and standard deviation each time the objective function is called by the optimization algorithm, the samples will be clustered around each design point, like islands in a lake. Unfortunately samples may be drawn far away from the islands, where precision is less good, when the metamodel is used during later iterations of the optimization algorithm.

For Approach 4 and Approach 5 to work, enough iterations of the optimization need to be made to enable the fitting of an accurate metamodel in the vicinity of the optimum. Since an LHS is performed for each sample used to fit the kriging model, the required number of model simulations equals the number of samples drawn for each LHS multiplied by the number of samples needed to fit an accurate metamodel. Consequently, the number of simulations will be quite high and as can be seen in the examples for Approach 2, quite accurate global metamodels can be fitted using the same number of simulations.

In these examples, it would seem that the allowed numbers of function calls is too small to create accurate metamodels for Approach 4 and Approach 5 since their hit-rates are not as good as in Approach 1. It is possible to reallocate the calls of the original function between the samples that are used to estimate the mean value and standard deviation, and the samples that are used to construct the kriging models. But if the number of function calls should remain the same, using more samples to construct the kriging models will result in fewer samples for estimating the mean value and standard deviation.

Approach 2 and Approach 3 seem more robust than Approach 4 and Approach 5 since all allowed calls of the original function for Approach 2 and Approach 3 are used to fit a kriging model, whereas the allowed calls are split between fitting a kriging model and estimating the mean value and standard deviation for Approach 4 and Approach 5.

Approach 6 performs better than the other methods for all of the tested problems, which indicates that it is possible to fit a metamodel iteratively during the optimization process and perform robust design optimization effectively. It is possible to increase the performance further by increasing the complexity of the approach, but it is important that the mechanisms are not too complicated. It is desirable to use an approach which is effective, robust and easy to use in real industrial problems.

VI. Conclusion The comparison of the different approaches highlights their differences and can be used as a guideline for

choosing an approach in other engineering problems. Since Approach 1 does not involve metamodels, it can be seen as a reference for the precision of the other approaches. The two analytical functions support the comparison since the analytically optimal point is known, whereas the aircraft system model demonstrates the performance of the approaches for engineering problems.

The approaches also suggest optimal parameters for the aircraft system model and consequently knowledge of the system is increased.

The sixth approach, which creates metamodels iteratively during the optimization process, outperforms the other approaches. It is also quite easy to use which means that it is a promising candidate for anyone which interested in performing robust design optimization. The other three approaches which create metamodels iteratively during the design process perform inadequately. When the metamodels are created after several iterations of the optimization process, the samples that are used to fit the metamodels will be placed in the vicinity of the optimum. The metamodels will as a result focus on reanimating the vicinity of the optimum accurately, which leads to good estimations when the metamodels are used during the later iterations of the optimization process. The drawback shown in the examples is that these approaches may need many samples to create accurate metamodels and that they therefore need to be carefully designed to get a desirable performance. The approaches which reanimate the mean value and standard deviation or the objective function perform worse than the other approaches that uses metamodels and are therefore deemed unsuitable for robust design optimization.

The common approach of creating a global metamodel of the original model and performing an RDO on the metamodel also performs well. The fitting of a global metamodel has two other advantages. The global metamodel


13

can be used after the optimization for other analyses. The metamodels fitted during the optimization are accurate in a smaller part of the design space. Moreover, the original model might be too expensive to enable tens of simulations, leaving the fitting of a global metamodel as the only realistic option.

Naturally, it is up to the system engineer to decide which method is best for a particular problem. However, it seems appropriate to use a method which involves metamodels for RDO of computationally expensive models, to avoid too tedious calculations.

Acknowledgments The research performed in this paper has received founding by the European Communitys Seventh Framework

Program under grant agreement no. 234344 (www.crescendo-fp7.eu).

References 1Lnn, D., Robust design - Accounting for uncertainties in engineering, Linkping Studies in Science and Technology.,

Thesis No 1389, Linkping University, Linkping, Sweden 2008. 2Kazemi, M. Metamodel-Based Optimization for Problems With Expensive Objective and Constraint Functions, Journal of

Mechanical Design, Vol. 133, January 2011. 3Padulo, M., Computational Engineering Design under uncertainty An aircraft conceptual design perspective, Cranfield

University, PhD Thesis, July 2009. 4Mavris, D.N., DeLaurentis, D.A., Bandte, O. and Hale, M.A.,A Stochastic Approach to Multi-Disciplinary Aircraft

Analysis and Design, AIAA 98-0912, 1998. 5Andersson J., Multiobjective Optimization in Engineering Design - Applications to Fluid Power Systems, Dissertation,

Linkping studies in Science and Technology, Dissertation No. 675, Linkping University, Linkping, Sweden, 2001. 6Beckwith, T. G., Marangoni, R.D. and Lienhard V, J.H., Mechanical Measurements, sixth edition, Pearson Prentice Hall,

Upper Saddle River, NJ, 2009, Chaps. 3. 7Beyer, H-G., and Sendhoff, B., Robust Optimization A comprehensive survey, Computer Methods in Applied

Mechanics and Engineering, Vol. 196, 2007, pp. 3190-3218. 8Park, H-S., and Dang, X-P., Structural optimization based on CAD-CAE integration and metamodeling techniques,

Computer-Aided Design, Vol. 42, 2010, pp. 889-902. 9Wang, G., and Shan, S., Review of Metamodeling Techniques in Support of Engineering Design Optimization, Journal of

Mechanical Design, Vol. 129, No. 2, April 2007, pp. 370-380. 10Duvigneau, R., and Praveen, C., Meta-Modeling for Robust Design and Multi-Level Optimization, 42nd AAAF Congress

on Applied Aerodynamics, Sophia-Antipolis, France, March 2007 11Jin, R., Du, X., and Chen, W., The use of metamodeling techniques for optimization under uncertainty, Journal of

Structural & Multidisciplinary Optimization, Vol. 25, No. 2, 2003, pp. 99-116. 12Jin, R., Chen, W., and Simpson, T. W., Comparative Studies of Metamodeling Techniques under Multiple Modeling

Criteria, Journal of Structural & Multidisciplinary Optimization, Vol. 23, No. 1, 2001, pp. 1-13. 13Shan, S., and Wang, G., Metamodeling for High Dimensional Simulation-Based Design Problems, Journal of Mechanical

Design, Vol. 132, May 2010. 14Tarkian, M., Persson, J. A., lvander, J., and Feng, X., Multidisciplinary Design Optimization of Modular Industrial

Robots, Proceedings of the ASME 2011 International Design Engineering Technical Conferences & Computers and information in Engineering Conference, 2011, pp. 1-10.

15Martin, J. D. and Simpson, T. W., A methodology to manage uncertainty during system-level conceptual design, Proceedings of IDETC/CIE 2005 ASME 2005 International Design Engineering Technical Conferences & Computers and information in Engineering Conference, 2005, pp. 1-11.

16Martin, J. D. and Simpson, T. W., A Monte Carlo Simulation of the Kriging Model, 10th AIAA/ISSMO Symposium on Multidiscipilinary Analysis and Optimization, AIAA-004-4483, 2004.

17Shao, X. Y., Chu, X. Z., Gao. L., and Qui, H. B., A novel method to manage uncertainty for multidisciplinary design and optimization in conceptual design, Advanced Materials Research, Vol. 44-46, 2008, pp. 225-232.

18McKay, M.D., Beckman, R.J. and Conover, W.J., A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code, Technometrics, Vol. 21. No. 2 , 1979, pp. 239-245.

19Persson, J. A., and lvander, J., Comparison of Sampling Methods for a Dynamic Pressure Regulator, AIAA-2011-1205, Proceedings of the 49th AIAA Aerospace Sciences Meeting Including the New Horizons Forum and Aerospace Exposition, AIAA, Orlando, FA, 4-7 Jan. 2011.

20lvander J., and Krus P., "Optimizing the Optimization - A Method for Comparison of Optimization Algorithms", proceedings of the 2nd AIAA Multidisciplinary Design Optimization Specialists Conference, Newport, RI, USA, May 1-4, 2006.

21Persson, J. A., and lvander, J., A Modified Complex Algorithm Applied to Robust Design Optimization, AIAA-2011-1205, Proceedings of the 13th AIAA Non-Deterministic Approaches Conference, AIAA, Denver, CO, 4-7 Apr. 2011.

22Saltelli, A., Chan, K. and Scott, E.M., Sensitivity Analysis, John Wiley & Sons, Chichester, West Sussex, 2000, pp. 10. 23Steinkellner, S. and Krus, P., "Balancing Uncertainties in Aircraft System Simulation Models", 2nd European Air & Space

Conference CEAS, Manchester, United Kingdom, 2009.


14

24Forrester, A. I. J., Sbester, A. and Keane A, J., Engineering Design via Surrogate Modelling: A Practical Guide, John Wiley & Sons Ltd, Chichester, West Sussex, United Kingdom, 2008, Chaps. 1.

25Simpson, T. W., Peplinski, J., Koch, P. N., and Allen, J. K., Metamodels for Computer-Based Engineering Design: Survey and Recommendations, Engineering with Computers, Vol. 17, No. 2, 2001, pp. 129-150.

26Krus, P. and Andersson, J., "Optimizing Optimization for Design Optimization", Proceedings of ASME Design Automation Conference , Chicago, USA, September 2-6, 2003.

27Lnn, D., Fyllingen, ., and Nilsson, L., An approach to robust optimization of impact problems using random samples and metamodelling, International Journal of Impact Engineering, Vol. 37, 2010, pp. 723-734.

28Aspenberg, D., Jergeus, J., and Nilsson, L., Robust optimization of front members in a full frontal car impact, Engineering Optimization, 2012, DOI: 10.1080/0305215X.2012.669380.

Documents

AIAA_RDO