An adaptive RBF-HDMR modeling approach under limited ...rrntucorplab.ntu.edu.sg/Research/DACS/Documents/publications/CR… · An adaptive RBF-HDMR modeling approach under limited

Struct Multidisc Optimhttps://doi.org/10.1007/s00158-017-1807-0

RESEARCH PAPER

An adaptive RBF-HDMR modeling approach under limitedcomputational budget

Haitao Liu1 · Jaime-Rubio Hervas2 ·Yew-Soon Ong2,3 · Jianfei Cai2 ·Yi Wang4

Received: 28 February 2017 / Revised: 3 June 2017 / Accepted: 3 September 2017© Springer-Verlag GmbH Germany 2017

Abstract The metamodel-based high-dimensional modelrepresentation (e.g., RBF-HDMR) has recently been provento be very promising for modeling high dimensional func-tions. A frequently encountered scenario in practical engi-neering problems is the need of building accurate modelsunder limited computational budget. In this context, theoriginal RBF-HDMR approach may be intractable due tothe independent and successive treatment of the componentfunctions, which translates in a lack of knowledge on whenthe modeling process will stop and how many points (sim-ulations) it will cost. This article proposes an adaptive and

� Haitao [email protected]

Jaime-Rubio [email protected]

Yew-Soon [email protected]

Jianfei [email protected]

Yi [email protected]

1 Rolls-Royce@NTU Corporate Laboratory, NanyangTechnological University, Singapore, 637460, Singapore

2 School of Computer Science and Engineering, NanyangTechnological University, Singapore, 639798, Singapore

3 Data Science and Artificial Intelligence Research Center,Nanyang Technological University, Singapore,639798 Singapore

4 Applied Technology Group, Rolls-Royce Singapore, 6 SeletarAerospace Rise, Singapore, 797575 Singapore

tractable RBF-HDMR (ARBF-HDMR) modeling frame-work. Given a total of Nmax points, it first uses Nini pointsto build an initial RBF-HDMR model for capturing thecharacteristics of the target function f , and then keeps adap-tively identifying, sampling and modeling the potential cutswith the remaining Nmax−Nini points. For the second-orderARBF-HDMR, Nini ∈ [2n + 2, 2n2 + 2] not only dependson the dimensionality n but also on the characteristics of f .Numerical results on nine cases with up to 30 dimensionsreveal that the proposed approach provides more accuratepredictions than the original RBF-HDMR with the samecomputational budget, and the version that uses the maximinsampling criterion and the best-model strategy is a recom-mended choice. Moreover, the second-order ARBF-HDMRmodel significantly outperforms the first-order model; how-ever, if the computational budget is strictly limited (e.g.,2n + 1 < Nmax � 2n2 + 2), the first-order model becomesa better choice. Finally, it is noteworthy that the proposedmodeling framework can work with other metamodelingtechniques.

Keywords Metamodeling · Adaptive high dimensionalmodel representation · Limited computational budget ·Tractable process

1 Introduction

Metamodeling techniques have been extensively used inengineering design and optimization in order to alleviatecomputational costs. A complete review of typical meta-modeling techniques, e.g., Kriging, radial basis functions(RBF), moving least square (MLS), support vector regres-sion (SVM) and polynomial response (PR), can be found inWang and Shan (2007) and Razavi et al. (2012). Although

http://crossmark.crossref.org/dialog/?doi=10.1007/s00158-017-1807-0&domain=pdf

mailto:[email protected]





H. Liu et al.

being successful in various applications, it has been pointedout that in high dimensional scenarios, the accuracy andefficiency of these modeling techniques sharply decreasedue to the “curse of dimensionality” (Shan and Wang2010b). With the current increase of systems complexity,simulation-based engineering problems with large dimen-sionality (e.g., greater than 10) are frequently encountered.Therefore, a metamodeling technique that effectively tack-les high dimensionality is needed.

One way to alleviate the “curse of dimensionality” isto approximate the target function by a combination oflow-dimensional functions. In this context, Shan and Wang(2010b) reviewed and classified the strategy into two cat-egories: adaptive computation and additive models. Exam-ples of adaptive computation include projection pursuitregression (Friedman and Stuetzle 1981) and classificationand regression trees (CART) (Breiman et al. 1984); exam-ples of additive models include additive interactive regres-sion (Andrews and Whang 1990) and high-dimensionalmodel representation (HDMR). Here, we mainly focus onthe HDMR strategy.

The well-known HDMR strategy is developed from someseminal works by (Sobol 1993, 2003; Rabitz et al. 1999). Itfollows the theorem that for any integrable function, thereexists a unique high-dimensional model representation thatis of finite order and offers a hierarchical structure. In a sim-ilar spirit to Taylor expansion, the HDMR contains a familyof representations in which each reflects the individual andcorrelated contributions of the input variables to the outputresponse. Thus, it can act as a general set of quantitativemodel assessment and analysis tools. Thereafter, a family ofHDMRs with different characteristics was proposed for thepurpose of, for instance, sensitivity analysis, reliability anal-ysis and modeling (Rabitz and Alis 1999, Li et al. 2001a, b,2002, 2006, 2008; Tunga and Demiralp 2005, Chowdhuryand Rao 2009; Liu et al. 2016d).

Among these HDMRs, there are mainly two expan-sions, namely, ANOVA-HDMR and Cut-HDMR. In termsof sensitivity analysis, the ANOVA-HDMR (Rabitz and Alis1999, Li et al. 2002, 2006, 2008) is more suitable sinceit is designed for statistical purposes and is capable ofidentifying important variables and correlations. The maindrawback of ANOVA-HDMR is that it needs to computemany integrals. On the other hand, the Cut-HDMR (Li et al.2001a, b), which involves no integral operations, is an exactrepresentation of the target function f (x) by decomposingf (x) into a set of component functions on cut lines, cutplanes and cut hyperplanes passing through a user-definedcut point (also known as reference point). Therefore, interms of modeling, the Cut-HDMR is more attractive dueto the ease of implementation. The original Cut-HDMR,however, is incomplete since it only offers a check-uptable.

Shan and Wang (2010a) combined RBF with Cut-HDMRto construct a promising high-dimensional modelingapproach, termed RBF-HDMR. The schematic flowchartof the second-order RBF-HDMR is shown in Fig. 1. TheRBF-HDMR framework approximates the component func-tions on the cut lines and planes successively via RBFand an accompanying sampling strategy, and then com-bines all the component RBF models together to form acomplete model. In this metamodel based HDMR mod-eling framework, different effective sampling approaches,e.g., maximin sampling (Ulaganathan et al. 2016), DIRECT-based sampling (Huang et al. 2015) and LOLA-Voronoisampling (Ulaganathan et al. 2016; Cai et al. 2016), can beused for the accompanying sampling process; similarly, dif-ferent metamodeling techniques, e.g., Kriging (Tang et al.2013), gradient-based Kriging (Ulaganathan et al. 2016),enhanced RBF model (Cai et al. 2016), MLS (Li et al.2012) and SVR (Huang et al. 2015), can be employed forapproximating the component functions.

These metamodel-based HDMR approaches are promis-ing for high-dimensional modeling; however, they presenttwo important disadvantages when handling practical engi-neering problems, namely:

– The model construction cannot utilize random points.As shown in Fig. 1, the typical HDMR modeling pro-cess requires well-structured points sampled only oncut lines and planes. In practice, however, cases wherethere exists some data lying outside the cut lines andplanes are frequently encountered. To address this issue,Liu et al. (2017b) proposed a generalized RBF-HDMR(GRBF-HDMR) model by using the error-allocationstrategy to utilize random data;

– The model construction is passive and intractable. Asshown in Fig. 1, current metamodel-based HDMRslocally approximate the component functions, i.e., theyare modeled independently and successively until con-vergence. It is intractable since there is a lack ofknowledge on when the process will stop and howmany points it will cost. It can be argued that themodeling process can be forcibly terminated when thecomputational budget is exhausted. But this will resultin an incomplete model, since some component func-tions may not have been approximated yet. Besides,due to the local view, the over-/under-sampling phe-nomenon often occurs on some cuts, and deterioratesthe prediction quality.

This article focuses on the second issue, and presentsan adaptive and tractable RBF-HDMR (ARBF-HDMR)approach to build an accurate model under limited com-putational budget, e.g., given Nmax points (simulations).The proposed approach adaptively refines the model in atractable and open way: it first builds an initial RBF-HDMR

An adaptive RBF-HDMR modeling approach under limited computational budget

Fig. 1 Schematic flowchart ofthe second-order RBF-HDMRmodeling process

model to have a global view on the characteristics of thetarget function, and then adaptively identifies, samples andmodels the potential cuts until the computational budget isexhausted.

The remaining of this article is organized as follows.Section 2 introduces the original RBF-HDMR modelingapproach. Section 3 presents the proposed ARBF-HDMRmodeling approach. Sections 4 and 5 assess the performanceof ARBF-HDMR through extensive numerical experiments.Finally, Section 6 provides some concluding remarks.

2 The original RBF-HDMR modeling approach

2.1 The RBF interpolation model

The RBF interpolation model is a family of models withthe same structure but distinguished by the choice of basisfunctions (Fang and Horstemeyer 2006). Suppose that f (x)is the target function to be approximated in the design spaceD ∈ R

n. Given a set of observed points X = {x1, · · · , xm}Tand their output responses y = {f (x1), · · · , f (xm)}T , theRBF model f (x) is

f (x) =m∑

i=1

λiφ (‖x − xi‖) +u∑

j=1

μjpj (x) (1)

where φ is a basis function, λi is the weight for the i-thobserved point, p is a linear polynomial function, μj is thecoefficient for the j -th polynomial term, and u is the numberof polynomials. A widely used basis function has the multi-quadric form

φ(r) =√

(r2 + s2) (2)

where r = ‖x − xi‖ is the Euclidean distance betweentwo points, and s > 0 is a user-defined shape parame-ter. Other forms of φ such as the Gaussian basis functionφ(r) = e−r2/s2

and the cubic basis function φ(r) = (r +s)3

may also be used.To determine the coefficients λ = {λ1, · · · , λm}T and

μ = {μ1, · · · , μu}T , we adopt the interpolation conditions

f (xi ) = f (xi ), 1 ≤ i ≤ m (3)

and the additional orthogonality conditionsm∑

i=1

λipj (xi ) = 0, 1 ≤ j ≤ u (4)

Thereafter, by defining Aik = φ(‖xi − xk‖) (1 ≤ i, k ≤ m)

and Pij = pj (xi ) (1 ≤ i ≤ m, 1 ≤ j ≤ u), the RBF modelis obtained by solving the following (m+u)×(m+u) linearsystem(

A PPT 0

)(λ

μ

)=

(y0

)(5)

The RBF model is found to be simple and effectivefor approximating complex functions from low to highdimensions (Liu et al. 2016b).

2.2 The RBF-HDMR model for high-dimensionalnonlinear problems

Let x = (x1, · · · , xn) ∈ [0, 1]n be an n-dimensionalpoint, a general HDMR model for the target function f (x)can be expressed in a hierarchical structure as (Rabitzand Alis 1999)

f (x) = f0 +n∑

i=1fi(xi) + ∑

1≤i<j≤n

fij (xi, xj ) + · · ·+ ∑

1≤i1<···<il≤n

fi1···il (xi1 , · · · , xil )

+f1···n(x1, · · · , xn)

(6)

H. Liu et al.

where the constant term f0 denotes the zeroth-order effecton f (x), the first-order term fi(xi) is the effect of the sin-gle variable xi on f (x), the second-order term fij (xi, xj )

is the correlated contribution of the two variables xi and xj

on f (x) after removing their individual effects, the higher-order terms reflect the effects of the increasing numbers ofcorrelated variables acting together on f (x), and the finalterm f1···n(x1, · · · , xn) denotes the residual effect of all thevariables acting together on f (x) after all the lower-ordercorrelations and individual effects have been removed.

For an n-dimensional target function, the number ofall possible component functions in a complete HDMRexpansion is

N =n∑

i=0

n!(n − i)!i! = 2n (7)

and the expansion is always exact (Rabitz et al. 1999). Itis observed that N dramatically increases with the dimen-sionality. Fortunately, for most physical systems, only thelow-order correlations among variables significantly influ-ence the output response (Rabitz and Alis 1999). In thiscontext, the second-order HDMR expansion is widely usedgiven that it provides an acceptable modeling accuracyfor most physical systems while significantly reducing therequired computing cost.

Among existing HDMRs (Li et al. 2001b; Tunga andDemiralp 2005; Wang et al. 2003), the Cut-HDMR (Rab-itz and Alis 1999) is found to have a simple structure andprovide a cost-efficient model with a similar accuracy toother HDMRs. Hence, the Cut-HDMR is chosen as the basisfor RBF-HDMR (Shan and Wang 2010a). The Cut-HDMRexpansion approximates along the cuts (cut lines, planes andhyperplanes) passing through a cut point in the domain. Bychoosing a cut point x0 = (x10 , · · · , xn0), the expressionsof the component functions in Cut-HDMR are given as

f0 = f (x0) (8)

fi(xi) = f (xi, xi0) − f0 (9)

fij (xi, xj ) = f (xi, xj , xij

0 )

−fi(xi) − fj (xj ) − f0 (10)

fijk(xi, xj , xk) = f (xi, xj , xk, xijk

0 ) − fij (xi, xj )

−fik(xi, xk) − fjk(xj , xk)

−fi(xi) − fj (xj ) − fk(xk) − f0 (11)

...

f1···n(x1, · · · , xn) = f (x) − f0 −∑

i

fi(xi)

−∑

ij

fij (xi, xj ) − · · · (12)

where xi0, xij

0 , and xijk

0 denote x0 without the elementsxi0 ; xi0 , xj0 ; and xi0 , xj0 , xk0 ; respectively. The points x0,

(xi, xi0) = (x10 , · · · , xi, · · · , xn0), and (xi, xj , x

ij

0 ) = (x10 ,

· · · , xi, · · · , xj , · · · , xn0), that lie on cut lines and planes,are referred as the zeroth-, first- and second-order points,respectively. Correspondingly, f (xi, xi

0) is the response

of f (x) at (xi, xi0), and f (xi, xj , x

ij

0 ) is the response of

f (x) at (xi, xj , xij

0 ). While fi(xi), which is different fromf (xi, xi

0), is the first-order component function response at xi

along the i-th cut line, and fij (xi, xj ) is the second-ordercomponent function response at (xi, xj ) on the i-j cut plane.

Equations (8)–(12) only offer a check-up table, and can-not be used for data interpolation in this current version.To have a complete and available model, Shan and Wang(2010a) proposed to use the RBF technique together withan accompanying sampling strategy to approximate thecomponent functions.

Some notations are defined before introducing the RBF-HDMR model. The point set Xi = {(xi, xi

0)}T contains thefirst-order points on the i-th cut line, the vector Xi = {xi}Tis the i-th column of Xi , and Yi = {fi(xi)}T is a vectorof the first-order component function responses calculatedby Eq. (9). Similarly, the set Xij = {(xi, xj , x

ij

0 )}T collectsthe second-order points on the i-j cut plane, the set Xij ={(xi, xj )}T picks the i-th and j -th columns of Xij , andYij = {fij (xi, xj )}T contains the second-order componentfunction responses computed by (10).

The second-order RBF-HDMR model, which is sufficientlyaccurate for most of the practical cases, is expressed as

f (x) ≈ f0 +n∑

i=1

fi(xi) +∑

1≤i<j≤n

fij (xi, xj )

≈ f0 +n∑

i=1

fi (xi) +∑

1≤i<j≤n

fij (xi, xj ) = f (x) (13)

where fi , a 1D RBF model built using Xi and Yi , is theapproximation of the first-order component function fi ;and fij , a 2D RBF model built using Xij and Yij , is theapproximation of the second-order component function fij .The second-order RBF-HDMR modeling process is con-ceptually illustrated in Fig. 1. It first approximates thefirst cut line until convergence, and then the remaining cutlines successively. If the second-order terms exist, it againapproximates the cut planes successively until convergence.The convergence of the RBF modeling process on each cutis solely measured by the relative prediction error at a testpoint.

The RBF-HDMR, which turns a black-box function intoan analytical functional form, presents a modeling frame-work which reveals the nonlinearity of the function withrespect to the single variables, the existence of high-orderterms, and the correlation of two variables in order to reduce


the required number of points for modeling. Moreover, thisinformation can further help RBF-HDMR to perform wellfor sensitivity analysis, decomposition, visualization, opti-mization, etc. It is to be noted that Shan and Wang (2011)efficiently extended the modeling process of RBF-HDMRfrom the second-order form to any higher order.

In this article, unless otherwise indicated, the RBF-HDMR model and the proposed ARBF-HDMR model areconstructed in the second-order fashion.

3 An adaptive RBF-HDMR modeling approach

As shown in Fig. 2, the proposed ARBF-HDMR model-ing approach is composed of three steps: (1) building aninitial RBF-HDMR model to capture the characteristicsof the target function; (2) identifying a potential cut; and(3) sampling a new point and refining the RBF model onthe potential cut. Steps (2) and (3) form an adaptive andtractable modeling framework that (a) adaptively and effi-ciently improves the prediction quality of the ARBF-HDMRmodel; and (b) allows to stop and export a complete modelat any time.

3.1 Construction of an initial RBF-HDMR model

The first step is to build an initial RBF-HDMR model forhaving a global view on the characteristics of the targetfunction, e.g., the linearity of the first-order componentfunctions, the existence of the second-order terms, and thecorrelations between any two variables. The global viewof the target function can help removing redundant terms,thus (1) saving computing resources and (2) making the

Fig. 2 Schematic flowchart of the ARBF-HDMR modeling process

subsequent modeling process more targeted. The initialRBF-HDMR model is suggested to be built with as fewpoints as possible such that the majority of the subsequentpoints are adaptively sampled, which is beneficial for thefinal model accuracy.

To this end, the initial RBF-HDMR model for the targetfunction f (x) in D ∈ [0, 1]n is constructed in a regular wayas follows:

1. The center point of the domain D is selected as the cutpoint x0 wherein xi0 = 0.5;

2. Two end points (x+i , xi

0) and (x−i , xi

0) along the i-th cut

line are added to fit the first-order RBF model fi . Notethat, in the unit space, we have x+

i = 1 and x−i = 0;

3. After building the first-order RBF-HDMR modelf (x) = f0 +∑n

i=1 fi (xi), a corner point (x+1 , · · · , x+

n )

is used to check for the existence of second-order terms.That is, if the relative error of f (i.e., |(f −f )/f |) at thisvalidation point is less than 0.1, the modeling processterminates; otherwise, it goes to step 4;

4. In the i −j cut plane, a validation point (x+i , x+

j , xij

0 ) isused to check for the correlation between the variablesxi and xj . If they are correlated, i.e., the relative errorof f is less than 0.1 at the validation point, four cornerpoints are added on the i-j cut plane to fit the second-order RBF model fij .

5. Export the initial RBF-HDMR model f (x) = f0 +∑ni=1 fi (xi) + ∑

1≤i<j≤n fij (xi, xj ).

Algorithm 1 provides a detailed description of the initialRBF-HDMR modeling process.

It is found that for an n-dimensional target function, thenumber of points Nini required to build the initial RBF-HDMR model satisfies

Nini ≤ 1 + 2n + 1 + 4n(n − 1)

2= 2n2 + 2 (14)

where the first term “1” represents the cut point in step 1, thesecond term “2n” represents the number of points sampledon n cut lines in step 2, the third term “1” represents thevalidation point to check for the existence of second-orderterms in step 3, and the final term “4n(n − 1)/2” representsthe number of points sampled on the cut planes if all theinput variables are correlated in step 4.

It is worth noting that the Nini value for the second-orderARBF-HDMR not only depends on the dimensionality butalso on the characteristics of the target function f (x), e.g.,the existence of second-order terms, and the correlationsamong input variables. If f (x) is relatively simple and hasno second-order terms, then Nini = 1 + 2n + 1 = 2n + 2;if f (x) is very complex, i.e., it exists second-order termsand any two variables are correlated, then Nini = 2n2 + 2;otherwise, we have 2n + 2 < Nini < 2n2 + 2. In prac-tice, since f (x) is a black-box function, Nini cannot be

H. Liu et al.

determined in advance. Equation (14) indicates that we needat most 2n2+2 points to accomplish the initial RBF-HDMRmodeling process. Therefore, to build a complete second-order ARBF-HDMR model, the allowed maximal numberof points Nmax should be greater than 2n2 + 2. Besides, aswill be discussed in Section 5.4.1, if the computational bud-get is strictly limited, e.g., Nmax is much less than 2n2 + 2,we prefer building the first-order ARBF-HDMR whereinNini = 1+2n due to the ignorance of the second-order terms.

3.2 Identification of potential cuts

After building the initial RBF-HDMR model, the secondstep is to identify potential cuts for the subsequent adap-tive modeling process. A cut is said to be potential if the

associated RBF model has large prediction errors or largeprediction uncertainty. Thus, improving the RBF model onthe potential cut is beneficial for the improvement of theoverall ARBF-HDMR model.

The identification process operates in two levels: (1) atthe system level, it identifies a potential system that has agreater impact on the overall accuracy of the RBF-HDMRmodel between the first-order RBF model system FI ={fi}1≤i≤n and the second-order RBF model system FII ={fij }1≤i<j≤n; (2) at the cut level, it identifies the poten-tial cut line (plane) on which the RBF model has the worstperformance among the identified model system.

Note that the identification is first conducted at thesystem level by treating the model systems FI and FII

separately. This is because: (1) these two kinds of RBFmodels usually have considerable differences in magnitudein terms of predictions, since the second-order functionsplay a residual role after removing the impact of the first-order functions; and (2) they have different dimensions andcharacteristics.

To assist the identification process, we first need to esti-mate the accuracies of the RBF models. Here, a well-knowngeneralized cross validation mean square error (GMSE) cri-terion, also known as leave-one-out (LOO) cross validationerror criterion, is employed to assess the model accuracy as

GMSE =√√√√ 1

m

m∑

i=1

e2LOO(xi ) (15)

where

eLOO(xi ) = f (xi ) − f −i (xi ) (16)

In (16), f −i is a RBF model built using all the sampledpoints except the i-th one. An accurate RBF model prefers asmall GMSE value. The GMSE value usually overestimatesthe real model error but is capable of successfully estimatingthe real model error with sufficient points (Viana et al. 2009;Liu et al. 2016c). Using (15), the global error of fi is esti-mated as GMSEi , and the global error of fij is as GMSEij .Given m sampled points, the RBF modeling process aug-mented with u polynomial terms is equivalent to solving a(m+u)×(m+u) linear system, see (5). To obtain the GMSEvalue by (15), we should remove one of the m sampledpoints each time and refit the RBF model. That is, we needto solve a (m+u−2)×(m+u−2) linear system m times. TheLU decomposition can be used to solve the linear system;however, the computational complexity associated with theGMSE calculation using this approach is O(m4), which isespecially time-consuming for cases with large sample sizesand high dimensions. For the original RBF model withoutpolynomial terms, Rippa (1999) presented a simple formulato speed up the LOO computation in (16). Here, for the RBF


model with polynomial terms, we derive a similar formulafor the fast evaluation of the LOO error.

Let the linear system of RBF be organized as

Bc = f (17)

where

B =(

A PPT 0

), c =

(λ

μ

), f =

(y0

)

Then, the LOO error at the observed point xi (1 ≤ i ≤ m)

can be calculated as

eLOO(xi ) = f (xi ) − f −i (xi ) = ci

B−1ii

(18)

where ci is the i-th element of the interpolation coefficientvector c, and B−1

ii is the i-th diagonal element of the inverseof the interpolation matrix B. For calculating the LOO errorsat m sampled points, we need to compute the inverse ofthe matrix B only once, which results in the computationalcomplexity of O(m3).

At the system level, after obtaining the GMSE values ofthe RBF models, according to the short board effect, theoverall accuracy of system FI or FII is represented by theaccuracy of the worst member. That is,

{GMSEI = max{GMSEi}, 1 ≤ i ≤ n

GMSEII = max{GMSEij }, 1 ≤ i < j ≤ n(19)

If GMSEI > GMSEII , the first-order model system FI issaid to be the potential system, since the less accurate FI has agreater impact on the overall accuracy of the ARBF-HDMR.Hence, a new point is added on one of the cut lines to quicklyimprove the overall accuracy of the ARBF-HDMR. Oppo-sitely, if GMSEI < GMSEII , then the second-order resid-ual model system FII has a greater impact on the overallaccuracy of the ARBF-HDMR. Consequently, a new pointis added on one of the correlated cut planes.

Next, at the cut level, the potential cut among the potentialsystem is identified. Assume that GMSEI > GMSEII , thena score function is employed to rank the cut lines as

Si = GMSEi∑i GMSEi

+ ρi∑i ρi

, 1 ≤ i ≤ n (20)

where the first term of the right-hand side is the normalizedGMSE value of fi , and ρi = 1/mi in the second term isthe sample density on the i-th cut line where there are mi

first-order points. It is found that the first term is a localexploitation term that draws attention to the cut lines wherethe RBF models have larger prediction errors; whereas thesecond term is a global exploration term over the systemthat draws attention to the cut lines with a lower sample den-sity (i.e., higher uncertainty). The global exploration term

Fig. 3 A case showing the necessity of the global exploration term

can help avoid missing some undetected features on a cutline. For instance, Fig. 3 illustrates a case where the originalRBF-HDMR process asserts that this cut line has passed thelinearity check and will not visit it again. Actually, the maincharacteristics of the output response along this line havenot been revealed yet. By considering the global explorationterm, this kind of cut line can be revisited.

Similarly, if GMSEII > GMSEI , the score of the i-j cutplane can be calculated as

Sij = GMSEij∑i,j GMSEij

+ ρij∑i,j ρij

, 1 ≤ i < j ≤ n (21)

where the sample density on the i-j cut plane with mij

points is as ρij = 1/mij .

3.3 Effective sampling and modeling on potential cuts

After obtaining a potential cut, a new point should be addedto refit the RBF model on it. To further improve the predic-tion quality, this section investigates effective sampling andmodeling strategies on potential cuts.

3.3.1 Sampling on potential cuts

Given that the ARBF-HDMR modeling process is con-ducted in a sequential manner, a sequential samplingapproach is recommended to sample a new point xnew onthe potential cut. Current sequential sampling approachescan be classified into two categories: space-filling sequen-tial sampling strategy (Johnson et al. 1990; Crombecq et al.2011b; Liu et al. 2015) which generates points to fill theentire domain evenly, and adaptive sequential samplingstrategy (Crombecq et al. 2011a; Xu et al. 2014; Liu et al.2016a, 2017a) which purposefully generates more points inregions where the model yields large prediction errors.

H. Liu et al.

Among the space-filling sequential sampling approaches,the well-known maximin sampling approach (Johnson et al.1990) is employed. This approach selects a new point as theone that lies farthest away from the m existing points. Thatis,

xnew = arg maxx

mini

‖x − xi‖, 1 ≤ i ≤ m (22)

where xi is an existing point on the potential cut.Among the adaptive sequential sampling approaches, the

simple and effective CV-Voronoi sampling approach (Xuet al. 2014) is introduced. The CV-Voronoi first employs theVoronoi diagram algorithm to partition the entire domaininto a set of Voronoi cells as

Ci = {x ∈ D, | ‖x − xi‖≤‖x − xj‖}, 1 ≤ i �= j ≤ m

(23)

Thereafter, the cross-validation approach is adopted to esti-mate the prediction error of each cell, and the one with thelargest error is denoted as the sensitive cell

Csensitive = arg maxCi

e(Ci) (24)

where e(Ci) = |eLOO(xi )| =| f (xi ) − f −i (xi ) |. Finally,the new point xi is selected in the sensitive cell as

xnew = arg maxx∈Csensitive

‖x − xsensitive‖ (25)

To investigate which kind of sampling strategy is moreeffective for the ARBF-HDMR, the maximin samplingapproach and the CV-Voronoi sampling approach are usedin this article.

3.3.2 Modeling on potential cuts

It is observed that the quality of the component RBF mod-els has a great impact on the performance of the finalARBF-HDMR model. It is known that the RBF modelswith different basis functions yield different predictions,and the component functions in HDMR may have differ-ent characteristics. Hence, rather than building RBF modelswith a fixed basis function for all the component func-tions, we employ a best-model strategy to select the bestRBF model with an appropriate basis function for eachcomponent function.

For a component function, K RBF models using K dif-ferent basis functions are first constructed to form a pool ofRBF models as

F = {f 1φ1

, · · · , f KφK

} (26)

The best-model strategy uses the GMSE criterion to evalu-ate the accuracies of the K RBF model, and selects the onewith the smallest GMSE

f b(x) = f u(x) ∈ F (27)

where u = arg min1≤u≤K GMSEu.Through the best-model strategy, the component func-

tions with different characteristics may be separately mod-eled by the RBF models with different basis functions,thus yielding more accurate predictions. Note that the best-model strategy is employed to build the RBF models in boththe initial RBF-HDMR process and the adaptive modelingprocess.

3.4 Description of the ARBF-HDMR approach

For the target function, after building an initial RBF-HDMRmodel, Algorithm 2 elaborates on the subsequent ARBF-HDMR modeling process under limited computational bud-get. Compared to the passive and intractable RBF-HDMRframework in Fig. 1, it is found that the proposed modelingframework is adaptive and tractable, since

– it keeps monitoring the potential cut on which the RBFmodel has the poorest performance, thus effectivelyimproving the overall model accuracy;

– in addition, the modeling framework runs in an openway so that it can stop and export a complete model atany time, which is suitable for practical problems withlimited computational budget.

In Algorithm 2, the sampling criterion could be eitherthe maximin sampling criterion or the CV-Voronoi sam-pling criterion. Both of the two criteria obtain new pointsby solving an auxiliary optimization problem (Johnson et al.1990; Xu et al. 2014). It is found that for ARBF-HDMR,the cuts simply lie in a 1D or 2D space. Hence, to query thebest new point using the two sampling criteria, the auxiliaryoptimization problem is solved based on 1000 uniform can-didate points on a cut line, and 100 × 100 grid candidatepoints on a cut plane.

In the numerical experiments below, several versions ofthe ARBF-HDMR are adopted. The basic version, denotedas ARBFMM -HDMR, uses the maximin sampling approachto add new points on potential cuts and the RBF modelwith multi-quadric basis function to approximate the com-ponent functions. The version that adopts the CV-Voronoisampling approach and the RBF model with multi-quadricbasis function is denoted as ARBFCV V -HDMR. Finally, theversion that adopts the maximin sampling approach and thebest-model strategy is denoted as ARBFb

MM -HDMR.


4 An example of ARBF-HDMR

We employ a 3D analytical function

f (x) = x22 + x1x3 + x1 − 4, x ∈ [0, 1]3 (28)

to illustrate the ARBF-HDMR modeling process. This func-tion has linear responses along variables x1 and x3, quadricresponses along x2, and the two variables x1 and x3 arecorrelated.

In the comparison, generally, we need to set a com-putational budget Nmax for the studied problem. There-after, we run the RBF-HDMR and ARBF-HDMR modelingprocesses and evaluate their final prediction accuracies,

respectively. But as stated before, the RBF-HDMR model-ing process cannot stop with an arbitrary Nmax ; otherwise,it will produce an incomplete model. Hence, for the purposeof comparison, we first run the RBF-HDMR modeling pro-cess till convergence, and then use the number of points inRBF-HDMR as Nmax for ARBF-HDMR.

The RBF-HDMR model for this function is constructedin a regular way following Liu et al. (2017b). The componentRBF models are built with the multi-quadric basis function,and the related shape parameter s is calculated by an empiricalformula in Hardy (1971). It is found that the RBF-HDMRrequires 21 points to complete the modeling process.

Thus, to be fair, we set Nmax = 21 and use the proposedapproach to build the ARBF-HDMR models (ARBFMM -HDMR, ARBFCV V -HDMR and ARBFb

MM -HDMR). Thefirst fourteen points are used to build the initial RBF-HDMRmodel, and the remaining seven points are adaptively sam-pled on identified potential cuts. Note that for ARBFb

MM -HDMR, the best-model strategy selects the best RBF modelfor each component function from three kinds of RBF mod-els built with the multi-quadric, Gaussian and cubic basisfunctions, respectively.

Table 1 shows the modeling results of the RBF-HDMRand three ARBF-HDMR models for the test function. Here,two global criteria (R2 and relative average absolute error(RAAE)) and a local criterion (relative maximum absoluteerror (RMAE)) are employed to assess the modeling perfor-mance. The details of these three criteria will be providedin the next section. Note that the best results, i.e., the largestR2 value and the smallest RAAE and RMAE values, aremarked in bold.

It is observed that, due to the adaptive modeling frame-work, all the three ARBF-HDMR models perform muchbetter than the original RBF-HDMR model using the samenumber of points in terms of both global and local criteria.Among the three ARBF-HDMR models, ARBFb

MM -HDMRprovides much more accurate predictions, and ARBFCV V -HDMR has a slightly better performance than ARBFMM -HDMR.

Figure 4 depicts the component RBF models in the RBF-HDMR and three ARBF-HDMR modeling processes onthree cut lines and the correlated cut plane for the 3D case.It is observed that the RBF-HDMR puts almost half of thepoints on the second cut line to build the RBF model f2.On the other hand, the ARBF-HDMR models identify thatthe correlated cut plane is also relevant and should be con-sidered in order to efficiently improve the overall modelaccuracy. Thus, compared to the RBF-HDMR, the ARBF-HDMR models sample four more points on the cut plane,placing only six points on the second cut line. The testresults reveal that, because of the local view, over-sampling(for f2) and under-sampling (for f13) occur in the RBF-HDMR, whereas the ARBF-HDMR models determine the

H. Liu et al.

Table 1 The modeling resultsof RBF-HDMR and threeARBF-HDMR models for the3D test case

Model R2 RAAE RMAE Nmax

RBF-HDMR 9.9837E-01 3.4169E-02 8.2231E-02 21

ARBFMM -HDMR 9.9965E-01 1.3972E-02 6.8501E-02 14+7

ARBFCV V -HDMR 9.9968e-01 1.3299E-02 6.6558E-02 14+7

ARBFbMM -HDMR 9.9996E − 01 4.9381E − 03 2.3151E − 02 14+7

sample locations more reasonably from a global view, thusleading to more accurate predictions.

In addition, among the three ARBF-HDMR models, theARBFCV V -HDMR places points in a slightly different man-ner from that of the ARBFMM -HDMR, which results ina slightly better performance. The ARBFb

MM -HDMR andARBFMM -HDMR have the same sample distribution; how-ever, the ARBFb

MM -HDMR builds f1 and f3 using themulti-quadric basis function, while it builds f2 and f13

using the cubic basis function. It is observed that the best-model strategy significantly improves the prediction quality.

Furthermore, Fig. 5 shows the track of the 21 points inthe RBF-HDMR and three ARBF-HDMR models, respec-tively. In this figure, the horizontal axis represents the pointnumber and the vertical axis is the identity of the cut line(plane). For example, along the horizontal axis, the number“0” represents the center point of the domain, and the num-ber “2” indicates the second point sampled in the modelingprocess; along the vertical axis, the number “2” means thesecond cut line with variable x2, and the number “13” meansthe cut plane with variables x1 and x3.

It is observed that the RBF-HDMR samples the cuts inde-pendently and successively. It first samples two points inthe first cut line because of the linearity, then nine pointsin the second cut line, then two points in the third cut

line, and finally four points in the correlated cut plane. Therest of the points are the validation points to check for theexistence of second-order terms and the correlation of twovariables.

In the ARBF-HDMR modeling processes, the black cir-cles represent the initial RBF-HDMR modeling processthat captures the characteristics of the target function,whereas the red circles represent the subsequent adap-tive modeling process. It is observed that after obtainingthe initial RBF-HDMR model, the ARBFMM -HDMR andARBFCV V -HDMR identify the potential of the second cutline and sample two points there. As f2 improves, the cor-related cut plane becomes the potential one and is sampledwith two points. Then, the two models switch back to thesecond cut line, and finally they switch again to the corre-lated cut plane. On the other hand, the ARBFb

MM -HDMRfirst identifies the second cut line as the potential one andsamples three points, and then captures the correlated cutplane to sample the rest of the four points. It is found that theARBF-HDMRs determine the potential cuts in a dynamicway, which helps them outperform the RBF-HDMR. Here,“dynamic” means that, unlike the original RBF-HDMR thatsamples the cuts successively, the ARBF-HDMRs identifyand sample a potential cut in each iteration via the erroranalysis described in Section 3.2.

Fig. 4 The component RBFmodels in the RBF-HDMR andthree ARBF-HDMR modelingprocesses on three cut lines andthe correlated cut plane for the3D test case 0 0.5 1

-1

0

1

0 0.5 1-1

0

1

0 0.5 1-0.5

0

0.5

0 0.5 10

0.5

1

0 0.5 1-1

0

1

0 0.5 1-1

0

1

0 0.5 1-0.5

0

0.5

0 0.5 10

0.5

1

0 0.5 1-1

0

1

0 0.5 1-1

0

1

0 0.5 1-0.5

0

0.5

0 0.5 10

0.5

1

0 0.5 1-1

0

1

0 0.5 1-1

0

1

0 0.5 1-0.5

0

0.5

0 0.5 10

0.5

1


Fig. 5 The track of the 21points sampled in the a RBF-HDMR, b ARBFMM -HDMR, cARBFCV V -HDMR, and dARBFb

MM -HDMR modelingprocesses, respectively. In thethree ARBF-HDMRs, the blackcircles represent the initialRBF-HDMR modeling process,whereas the red circles representthe subsequent adaptivemodeling process

2 4 6 8 10 12 14 16 18 200123

121323

123

2 4 6 8 10 12 14 16 18 200123

121323

123

2 4 6 8 10 12 14 16 18 200123

121323

123

2 4 6 8 10 12 14 16 18 200123

121323

123

a

b

c

d

5 Additional numerical experiments

5.1 Test cases

To comprehensively assess the performance of the ARBF-HDMR, we employ five benchmark functions F1-F5 (Shanand Wang 2010a) and four engineering examples E1-E4, theexpressions of which are provided in the Appendix.

The first 5D engineering example (E1) is the design ofa direct methanol fuel cell system, in which methanol isused as the fuel to generate electricity via reaction with oxy-gen in the air (Yang and Xue 2015). We attempt to modelthe semi-empirical output voltage model of a specific fuelcell system, which is influenced by the current density I

(A/cm2), temperature T (K), methanol concentration CME

(M), methanol flow rate FME (ccm), and air flow rate FAIR

(ccm).The second 8D engineering example (E2) is a model

that describes the flow of water through a borehole thatis drilled from the ground surface through two aquifers(Morris et al. 1993). The design variables include the radiusof the borehole rw (m), radius of influence r (m), trans-missivity of upper aquifer Tu (m2/y), potentiometric headof upper aquifer Hu (m), transmissivity of lower aquifer Tl

(m2/y), potentiometric head of lower aquifer Hl (m), lengthof borehole L (m), and hydraulic conductivity of boreholeKw (m/y).

The third 10D engineering example (E3) is a conceptuallevel estimate of the weight of a light aircraft wing (For-rester et al. 2008). The design variables include the wingarea Sw (ft2), weight of fuel in the ring Wf w (lb), aspectratio A, quarter-chord sweep � (deg), dynamic pressure atcruise q (lb/ft2), taper ratio λ, aerofoil thickness to chordratio tc, ultimate load factor Nz, flight design gross weightWdg (lb), and paint weight Wp (lb/ft2).

The last 30D engineering example (E4) attempts tomodel the tip deflection δ of a ten-stepped cantilever beamwith a P = 50 kN force in the tip and a material of E = 200GPa and σallow = 350 MPa (Cheng et al. 2015). The width bi

(m), height hi (m) and length li (m) of each step are selectedas design variables. This problem presents a high level ofcomplexity as the global optimum is unknown.

5.2 Performance criteria

In the testing, two commonly used global performance crite-ria and a local performance criterion are employed to assessthe performance of the ARBF-HDMR.

H. Liu et al.

(1) R2

R2 = 1 −∑t

i=1

[f (xi ) − f (xi )

]2

∑ti=1

[f (xi ) − f

]2(29)

where xi is one of the t validation points and f isthe average response over the t points. The closer thevalue of R2 approaches one, the more accurate themodel is.

(2) RAAE

RAAE =∑t

i=1 | f (xi ) − f (xi ) |t × STD

(30)

where STD stands for standard deviation of the func-tion responses at t validation points. An accuratemodel prefers a small RAAE value.

(3) RMAE

RMAE = max1≤i≤t | f (xi ) − f (xi ) |STD

(31)

This local metric measures the maximal prediction error in thedesign space. An accurate model prefers a small RMAE value.

In the following numerical experiments, the above threecriteria are calculated using t = 5000 validation pointsgenerated by the Matlab routine lhsdesign.

5.3 Results and discussions

In the numerical experiments, the RBF-HDMR model isfirst built for each test case, and then the required num-ber of points acts as the computational budget Nmax for theARBF-HDMR. Table 2 provides the modeling results of theRBF-HDMR and three ARBF-HDMR models for the fivebenchmark functions and four engineering examples. Thebest results are marked in bold.

5.3.1 ARBF-HDMR vs RBF-HDMR

It is observed that, in terms of global and local criteria, theARBF-HDMR models perform much better than the RBF-HDMR for most of the test cases with the same number ofpoints, especially for F2, F4, F5, E1 and E3.

The impressive performance of the ARBF-HDMRsdirectly comes from the adaptive and tractable modelingframework. As stated before, the RBF-HDMR indepen-dently and successively handles the component functions,which induces over-/under-sampling on some cuts. In con-trast, the ARBF-HDMR determines which cut line or planeshould be improved by considering them from a globalperspective.

Figure 6 depicts the first-order component functions f1

and f2 for the E2 case. It is found that the function f2

has a higher nonlinearity than f1, but the responses off1 are several orders of magnitude larger than that of f2.Thus, the accuracy of the RBF model f1 more significantlyaffects the overall model accuracy. However, because ofthe local view, the RBF-HDMR cannot identify the impor-tance of f1, and samples only eight points on this cut linewhile ten points on the second nonlinear cut line. In con-trast, because of the global view, the ARBF-HDMR alwaysfocuses on improving the RBF models on the potential cuts.For instance, by comparing the GMSE errors of f1 and f2,the ARBFMM -HDMR recognizes the importance of f1 andsamples thirteen points on this cut line while only threepoints for f2. It is worth noting that in ARBFMM -HDMR,the thirteen points on the first cut line are not sampledin successive iterations, but sampled when the cut line isidentified as a potential one.

In addition to the local view, the over-/under-samplingproblems in the RBF-HDMR are also caused by the fact thatthis approach uses a single test point to assess the accuracyof the component RBF model in order to determine whetherthe modeling should terminate or not. This point error esti-mation, however, is not a good representation of the RBFmodel accuracy. Oppositely, the ARBF-HDMR employs themore accurate and robust GMSE criterion to estimate theRBF model accuracy.

5.3.2 Comparison among three ARBF-HDMR models

All the three ARBF-HDMR models follow the adaptiveand tractable modeling framework. They only differ in theemployed sampling and modeling strategies to approximatethe component functions on potential cuts.

The impact of different sampling strategies is first inves-tigated by comparing the results of ARBFMM -HDMRand ARBFCV V -HDMR. The test results show that theARBFMM -HDMR outperforms the ARBFCV V -HDMR infive out of the nine cases. Figure 7 shows the convergencecurves of ARBFMM -HDMR and ARBFCV V -HDMR for theF2 case. It is observed that in the early stage, the two ARBF-HDMR models converge similarly; in the middle stage, theARBFCV V -HDMR converges faster; in the later stage, theaddition of new points by the ARBFCV V -HDMR, on thecontrary, deteriorates the predictions.

We observed that for the five cases where ARBFMM -HDMR outperforms ARBFCV V -HDMR, ARBFCV V -HDMR quickly reduces the GMSEI of FI such that, afterseveral iterations, FII has an opportunity to be selected asthe potential system because GMSEI < GMSEII ; at thispoint, the true error of FI , however, is still larger than thatof FII . The adaptive CVV sampling criterion is capable ofefficiently decreasing the GMSE value of the RBF model,


Table 2 The modeling resultsof the RBF-HDMR and threeARBF-HDMR models for thefive benchmark functions andfour engineering examples

f Model R2 RAAE RMAE Nmax

F1 RBF-HDMR 7.0371E-01 4.4924E-01 1.7920E+00 297ARBFMM -HDMR 7.9924E-01 3.5936E-01 1.5848E+00 202+95ARBFCV V -HDMR 9.6770E − 01 1.3836E − 01 1.0741E + 00 202+95ARBFb

MM -HDMR 8.5068E-01 3.1201E-01 1.3364E+00 202+95F2 RBF-HDMR 9.9934E-01 2.4157E-02 5.6471E-02 347

ARBFMM -HDMR 1.0000E + 00 1.2316E-03 1.7318E-02 202+145ARBFCV V -HDMR 9.9998E-01 3.3987E-03 1.9452E-02 202+145ARBFb

MM -HDMR 1.0000E + 00 6.5812E − 04 1.1853E − 02 202+145F3 RBF-HDMR 9.9027E-01 6.6834E-02 4.7703E-01 289

ARBFMM -HDMR 9.9745E − 01 3.3200E-02 3.2207E − 01 70+219ARBFCV V -HDMR 9.9626E-01 3.7116E-02 5.0237E-01 70+219ARBFb

MM -HDMR 9.9730E-01 3.3065E − 02 3.4684E-01 70+219

F4 RBF-HDMR 9.9828E-01 3.2779E-02 1.4543E-01 109ARBFMM -HDMR 9.9932E-01 2.1112E-02 9.5776E-02 70+39ARBFCV V -HDMR 9.9963E-01 1.5777E-02 5.7993E-02 70+39ARBFb

MM -HDMR 9.9994E − 01 6.1562E − 03 2.7543E − 02 70+39F5 RBF-HDMR 8.9448E-01 2.5644E-01 1.5709E+00 320

ARBFMM -HDMR 9.9101E-01 7.4883E-02 4.1593E-01 244+76ARBFCV V -HDMR 9.9730E-01 4.0927E-02 2.3815E-01 244+76ARBFb

MM -HDMR 9.9879E − 01 2.7317E − 02 1.5761E − 01 244+76E1 RBF-HDMR 9.8486E-01 5.3289E-02 7.4436E-01 223




MM -HDMR 9.9958E − 01 1.2755E − 02 1.4380E − 01 73+138E3 RBF-HDMR 9.6251E-01 1.2345E-01 1.2793E + 00 488

ARBFMM -HDMR 9.7695E-01 8.1148E-02 1.4351E+00 100+388ARBFCV V -HDMR 9.7175E-01 1.0123E-01 1.3733E+00 100+388ARBFb

MM -HDMR 9.7769E − 01 7.9365E − 02 1.4285E+00 100+388E4 RBF-HDMR 9.5039E-01 1.6071E-01 1.6545E+00 939

ARBFMM -HDMR 9.5924E-01 1.4646E-01 1.6198E + 00 599+340ARBFCV V -HDMR 9.3484E-01 1.8670E-01 1.6906E+00 599+340ARBFb

MM -HDMR 9.6371E − 01 1.3315E − 01 1.6315E+00 599+340

0 0.25 0.5 0.75 1-60

-40

-20

0

20

40

60

80

100

0 0.25 0.5 0.75 1-0.05

0

0.05

0.1

0.15

0.2a b

Fig. 6 The plots of a f1 and b f2 for the E2 case

200 250 300 3500

0.005

0.01

0.015

0.02

0.025

Fig. 7 The convergence curves of ARBFMM -HDMR and ARBFCV V -HDMR for the F2 case

H. Liu et al.

since it sequentially adds new points in regions with thelargest LOO error via (25). However, the true model errormay not so quickly decrease. As a result, the discrepancymay lead to the wrong identification of the potential system.For example, for the F2 case in Fig. 7, the potential cutsidentified by ARBFMM -HDMR and ARBFb

MM -HDMR arecut lines in all the iterations; whereas ARBFCV V -HDMRbegins to identify cut planes as potential cuts after 272 sim-ulations and, correspondingly, the convergence curve startsto increase. Therefore, the space-filling maximin samplingcriterion is recommended for ARBF-HDMR.

The impact of different modeling strategies is next inves-tigated by comparing the results of ARBFMM -HDMR andARBFb

MM -HDMR. It is observed that compared to thefixed RBF model used in ARBFMM -HDMR, the best-modelstrategy significantly improves the prediction quality ofARBFb

MM -HDMR in all cases in terms of RAAE and inseven cases in terms of RMAE. Taking the E1 case forexample, the initial RBF-HDMR identifies that five first-order component functions and a second-order componentfunction should be approximated. As a result, the finalARBFMM -HDMR model uses the fixed multi-quadric basisfunction to build all the six component RBF models; incontrast, the final ARBFb

MM -HDMR model uses the cubicbasis function to build four component RBF models andthe Gaussian basis function to build the remaining two RBFmodels, which leads to more accurate predictions.

5.4 Other discussions

Taking the ARBFbMM -HDMR as the best version of ARBF-

HDMR, this section discusses the impact of model order andthe running time in ARBF-HDMR implementations.

5.4.1 Impact of model order

Table 3 shows the modeling results of the first-order RBF-HDMR and ARBFb

MM -HDMR for the nine test cases. Notethat due to the ignorance of the second-order terms, the Nini

value for the first-order ARBFbMM -HDMR is fixed at 1+2n,

where “1” denotes the cut point and “2n” denotes the pointson n cut lines.

It is observed that the first-order ARBFbMM -HDMR still

outperforms the first-order RBF-HDMR. In terms of thetwo global criteria, the ARBFb

MM -HDMR performs betterfor all the nine cases. In terms of the local criterion, theARBFb

MM -HDMR performs better in seven out of the ninecases.

By comparing the results in Tables 2 and 3, it is observedthat the first-order RBF-HDMR model outperforms thesecond-order version for F2 and E3. Shan and Wang (2010a)attributed this behavior to the fact that the errors in RBFconstruction may be larger than the impact of higher-orderterms, and thus leads to the over-fitting of RBF mod-els. We believe that it is due to the passive RBF-HDMRmodeling framework that cannot identify the cuts to befocused on. Oppositely, due to the global view, the second-order ARBFb

MM -HDMR model always outperforms thefirst-order model.

It may be argued that the computational budget Nmax

is not the same for the first- and second-order ARBFbMM -

HDMR models in Tables 2 and 3. For this reason, we takeF2 and E3 for example, and run the first-order ARBFb

MM -HDMR modeling process until the number of points is thesame as that for the second-order model. Figure 8 depictsthe convergence histories of the first- and second-orderARBFb

MM -HDMR models for F2 and E3, respectively. It is

Table 3 The modeling resultsof the first-order RBF-HDMRand ARBFb

MM -HDMR for thefive benchmark functions andfour engineering examples

f Model R2 RAAE RMAE Nmax

F1 RBF-HDMR 6.3053E-01 4.8151E-01 3.4103E+00 91ARBFb

MM -HDMR 7.3624E − 01 3.9712E − 01 3.1155E + 00 21+70F2 RBF-HDMR 9.9930E-01 2.3865E-02 1.0429E-01 31

ARBFbMM -HDMR 9.9991E − 01 6.8295E − 03 7.6987E − 02 21+10

F3 RBF-HDMR 9.9023E-01 6.6980E-02 4.7617E-01 181ARBFb

MM -HDMR 9.9692E − 01 3.7582E − 02 3.0727E − 01 21+160F4 RBF-HDMR 9.9705E-01 4.2718E-02 2.0483E-01 53

ARBFbMM -HDMR 9.9860E − 01 2.8303E − 02 1.2525E − 01 21+32

F5 RBF-HDMR 8.4039E-01 3.5858E-01 1.1374E+00 49ARBFb


ARBFbMM -HDMR 9.9024E − 01 6.6024E − 02 7.1869E − 01 11+23

E2 RBF-HDMR 9.6144E − 01 1.3724E-01 1.1557E+00 47ARBFb

MM -HDMR 9.6142E-01 1.3690E − 01 1.1545E + 00 17+30E3 RBF-HDMR 9.4443E-01 1.1423E-01 1.7548E + 00 90

ARBFbMM -HDMR 9.4945E − 01 1.0789E − 01 1.8463E+00 21+69

E4 RBF-HDMR 7.8764E-01 3.0175E-01 3.6137E + 00 171

ARBFbMM -HDMR 7.8925E − 01 2.9988E − 01 3.6788E+00 61+110


0 100 200 300 4000.999

0.9992

0.9994

0.9996

0.9998

1

0 100 200 300 4000

0.005

0.01

0.015

0.02

0.025

0 100 200 300 4000

0.05

0.1

0.15

0 200 400 600

0

0.5

1

0 200 400 6000

0.2

0.4

0.6

0.8

0 200 400 6001

2

3

4

5

6

a

b

Fig. 8 The convergence histories of the first-order and second-order ARBFbMM -HDMR models for a F2 and b E3, respectively

found that in the early stage, the second-order ARBFbMM -

HDMR needs to spend more points on the second-orderterms, whereas the first-order ARBFb

MM -HDMR ignoresthem and focuses all its attention on the improvement of thefirst-order RBF models. Hence, the first-order model con-verges faster in this stage. Thereafter, as the modeling pro-cess evolves, the first-order ARBFb

MM -HDMR is incapableto further improve the model accuracy because of the loss ofsecond-order terms. The comparison results reveal that theconsideration of second-order terms enlarges the space ofcomponent functions such that the ARBFb

MM -HDMR candetermine the locations of points more effectively.

The above discussions offer the guidelines for the prac-tical use of ARBFb

MM -HDMR: the second-order versionis recommended if Nmax is large; on the other hand, thefirst-order version becomes a better choice if Nmax is strictlylimited, e.g., 2n + 1 < Nmax � 2n2 + 2.

5.4.2 Running time

Table 4 provides the running times of RBF-HDMR andARBFb

MM -HDMR for the nine cases to illustrate the modelingefficiency. All the numerical experiments are executed in a

Matlab environment with an Intel 3.40 GHz processor. It isfound that the ARBFb

MM -HDMR requires much more com-puting time than the RBF-HDMR, particularly for caseswith a large sample size and high dimensionality (e.g., E3and E4).

The running time of ARBFbMM -HDMR mainly comes

from the cross-validation process of each RBF model.Assume that any two input variables of the target func-tion are correlated and each RBF model has m sampledpoints, then we should refit the RBF models mn(n + 1)/2times in order to estimate the GMSE values. This becomesmore and more time-consuming with the increase of sam-ple size and dimensionality. To alleviate the computingcost, the GMSE can be alternatively computed in a parallelfashion.

It is worth noting that the ARBFbMM -HDMR model

is proposed for expensive simulation-based problems, inwhich a simulation may require several hours or even days(Mueller et al. 2013). Thus, in practice, the running timeof ARBFb

MM -HDMR is negligible compared to the timespent on simulations. Besides, though the running timeof ARBFb

MM -HDMR itself is much larger than that ofRBF-HDMR, the ARBFb

MM -HDMR makes a great saving

Table 4 The running times (s) of RBF-HDMR and ARBFbMM -HDMR for the five benchmark functions and four engineering examples

Model F1 F2 F3 F4 F5 E1 E2 E3 E4

RBF-HDMR 1.4 0.4 1.6 0.2 0.4 0.4 0.3 5.1 1.9ARBFb

MM -HDMR 120.2 184.6 70.5 11.3 78.8 91.6 66.6 531.7 604.2

H. Liu et al.

in the time spent on simulations. For example, the RBF-HDMR requires 939 points (simulations) to achieve themodel accuracy of RAAE = 0.1607 for the 30D E4 case,whereas the ARBFb

MM -HDMR only needs 694 points toachieve the same model accuracy.

6 Concluding remarks

This article proposed an adaptive and tractable RBF-HDMRmodeling approach for high-dimensional problems underlimited computational budget. The extensive numericalresults offer the following findings:

– Compared to the original RBF-HDMR, the ARBF-HDMRprovides more accurate predictions in terms of both theglobal and local criteria with the same number of points;

– Among the different versions of ARBF-HDMR, theversion using the maximin sampling criterion and thebest-model strategy is recommended;

– the second-order ARBFbMM -HDMR always outper-

forms the first-order version due to the enlarged spaceof component functions. However, if the computationalbudget is strictly limited, e.g., 2n + 1 < Nmax �2n2 + 2, the first-order model is better.

For a black-box target function, the main disadvantageof the proposed ARBF-HDMR is that the Nini value inthe second-order modeling process cannot be determined inadvance, since it depends on both the characteristics and thedimensionality of the target function.

In this article, the ARBF-HDMR uses the RBF modelto approximate the component functions. It is worth notingthat the proposed approach can use some other metamodeltypes (e.g., Kriging and SVR), and even effective ensemblemodeling strategies (Liu et al. 2016c; Goel et al. 2007; Acarand Rais-Rohani 2009) for approximation purposes.

Acknowledgements The majority of this work was finished before join-ing the Lab. We appreciate the support from the National Research Foun-dation (NRF) Singapore under the Corp Lab@University Scheme forcompleting the research. It is also partially supported by the Data Scienceand Artificial Intelligence Research Center (DSAIR) and the School ofComputer Science and Engineering at Nanyang Technological University.

Appendix

Tables 5 and 6 offer the expressions of the employedfive benchmark functions and four engineering examples,respectively.

Table 5 Five benchmark functions

ID Expression Range

F1 f (x) = ∑10i=1

[(ln(xi − 2))2 + (ln(10 − xi))

2] − (

�10i=1xi

)0.2xi ∈ [2.1, 9.9]

F2 f (x) = ∑10i=1 xi

(ci + ln xi

x1+···+x10

)xi ∈ [1e−6, 10]

F3 f (x) = ∑10i=1 exi

[ci + xi − ln

(∑10k=1 exk

)]xi ∈ [−10, 10]

F4 f (x) = x21 + x2

2 + x1x2 − 14x1 − 16x2 + (x3 − 10)2 + 4(x4 − 5)2 + (x5 − 3)2 xi ∈ [−10, 11]+2(x6 − 1)2 + 5x2

7 + 7(x8 − 11)2 + 2(x9 − 10)2 + (x10 − 7)2 + 45

F5 f (x) = ∑16i=1

∑16j=1 aij (x

2i + xi + 1)(x2

j + xj + 1) xi ∈ [0, 5]For F2 and F3:

c1≤i≤10 = −6.089, −17.164, −34.054, −5.914, −24.721, −14.986, −24.1, −10.708, −26.662, −22.179.

For F5:

[aij ]rows1−8 =

⎡

⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣

1 0 0 1 0 0 1 1 0 0 0 0 0 0 0 10 1 1 0 0 0 1 0 0 1 0 0 0 0 0 00 0 1 0 0 0 1 0 1 1 0 0 0 1 0 00 0 0 1 0 0 1 0 0 0 1 0 0 0 1 00 0 0 0 1 1 0 0 0 1 0 1 0 0 0 10 0 0 0 0 1 0 1 0 0 0 0 0 0 1 00 0 0 0 0 0 1 0 0 0 1 0 1 0 0 00 0 0 0 0 0 0 1 0 1 0 0 0 0 1 0

⎤

⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦

[aij ]rows9−16 =

⎡

⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣

0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 10 0 0 0 0 0 0 0 0 1 0 0 0 1 0 00 0 0 0 0 0 0 0 0 0 1 0 1 0 0 00 0 0 0 0 0 0 0 0 0 0 1 0 1 0 00 0 0 0 0 0 0 0 0 0 0 0 1 1 0 00 0 0 0 0 0 0 0 0 0 0 0 0 1 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 1 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1

⎤

⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦


Table 6 Four engineering examples

ID Expression Range

E1 fV = 1.21 − 3.7534 × 10−5T − 3.1534 × 10−4T lnCME

+6.62 × 10−5T lnFAIR − 0.7499 − 6.9897e

(916.91

T−4.6392

)

I

−[1.2658 × 105I 3 + 46196I 2 − 4281I − 0.4029T

−18.8094C2ME + 18.8094CME + 10.496] I ∈ [0.0003, 0.08], T ∈ [298, 343]

×[lnI − 3.9056 + 2.9582 CME ∈ [0.25, 2], FME ∈ [3.5, 5.5]×10−4

(lnCME + ln

(1 − 1

5.3466×107e(−5182.4/T )C2ME

I

))] FAIR ∈ [81.2, 140.8]

−[−1.2687 × 105I 3 − 46221I 2 + 4283.6I + 0.4033T

+18.818C2ME − 18.818CME − 10.572]

×[lnI − 3.8959 − 8.2402 × 10−4lnFAIR] + 31.583I 2lnFME

E2 rw ∈ [0.05, 0.15], r ∈ [100, 50000]ff low = 2πTu(Hu−Hl)

ln rrw

[1+ 2LTu

ln rrw

r2wKw

+ TuTl

] Tu ∈ [63070, 115600], Hu ∈ [990, 1110]

Tl ∈ [63.1, 116], Hl ∈ [700, 820]L ∈ [1120, 1680], K2 ∈ [9855, 12045]

E3 Sw ∈ [150, 200], Wf w ∈ [220, 300]A ∈ [6, 10], � ∈ [−10, 10]

fw = 0.036S0.758w W 0.0035

f w

(A

cos2�

)q0.006λ0.04 q ∈ [16, 15], λ ∈ [0.5, 1]

×(

100tccos�

)−0.3(NzWdg)0.49 + SwWp tc ∈ [0.08, 0.18], Nz ∈ [2.5, 6]

Wdg ∈ [1700, 2500], Wp ∈ [0.025, 0.08]E4 fδ = ∫ ld

0Px2

d

EIddxd + ∫ ld−1

0P(xd−1+ld )2

EId−1dxd−1 + · · · bi ∈ [0.01, 0.05]

+ ∫ l10

P(x1+l2+l3+···+ld )2

EI1dx1 hi ∈ [0.3, 0.65]

= P3E

∑di=1

[12

bih3i

((∑dj=i lj

)3 −(∑d

j=i+1 lj

)3)]

li ∈ [0.5, 1], 1 ≤ i ≤ 10

References

Acar E, Rais-Rohani M (2009) Ensemble of metamodels with opti-mized weight factors. Struct Multidiscip Optim 37(3):279–294

Andrews DW, Whang YJ (1990) Additive interactive regression mod-els: circumvention of the curse of dimensionality. EconometricTheory 6(4):466–479

Breiman L, Friedman J, Stone CJ, Olshen RA (1984) Classificationand regression trees. CRC press, Boca Raton

Cai X, Qiu H, Gao L, Yang P, Shao X (2016) An enhanced RBF-HDMR integrated with an adaptive sampling method for approx-imating high dimensional problems in engineering design. StructMultidiscip Optim 53(6):1209–1229

Cheng GH, Younis A, Hajikolaei KH, Wang GG (2015) Trust regionbased mode pursuing sampling method for global optimizationof high dimensional design problems. J Mech Des 137(2):021–407

Chowdhury R, Rao B (2009) Hybrid high dimensional model repre-sentation for reliability analysis. Comput Methods Appl Mech Eng198(5):753–765

Crombecq K, Gorissen D, Deschrijver D, Dhaene T (2011a) A novelhybrid sequential design strategy for global surrogate modeling ofcomputer experiments. SIAM J Sci Comput 33(4):1948–1974

Crombecq K, Laermans E, Dhaene T (2011b) Efficient space-fillingand non-collapsing sequential design strategies for simulation-based modeling. Eur J Oper Res 214(3):683–696

Fang H, Horstemeyer MF (2006) Global response approximation withradial basis functions. Eng Optim 38(4):407–424

Forrester A, Sobester A, Keane A (2008) Engineering design viasurrogate modelling: a practical guide. Wiley, Hoboken

Friedman JH, Stuetzle W (1981) Projection pursuit regression. J AmStat Assoc 76(376):817–823

Goel T, Haftka RT, Shyy W, Queipo NV (2007) Ensemble of surro-gates. Struct Multidiscip Optim 33(3):199–216

Hardy RL (1971) Multiquadric equations of topography and otherirregular surfaces. J Geophys Res 76(8):1905–1915

Huang Z, Qiu H, Zhao M, Cai X, Gao L (2015) An adaptive SVR-HDMR model for approximating high dimensional problems. EngComput 32(3):643–667

Johnson ME, Moore LM, Ylvisaker D (1990) Minimax and maximindistance designs. J Stat Plan Inference 26(2):131–148

Li E, Wang H, Li G (2012) High dimensional model representa-tion (HDMR) coupled intelligent sampling strategy for nonlinearproblems. Comput Phys Commun 183(9):1947–1955

Li G, Rosenthal C, Rabitz H (2001a) High dimensional model repre-sentations. J Phys Chem A 105(33):7765–7777

Li G, Wang SW, Rosenthal C, Rabitz H (2001b) High dimensionalmodel representations generated from low dimensional data sam-ples. i. mp-Cut-HDMR. J Math Chem 30(1):1–30

Li G, Wang SW, Rabitz H (2002) Practical approaches to constructRS-HDMR component functions. J Phys Chem A 106(37):8721–8733

Li G, Hu J, Wang SW, Georgopoulos PG, Schoendorf J, Rabitz H(2006) Random sampling-high dimensional model representation(RS-HDMR) and orthogonality of its different order componentfunctions. J Phys Chem A 110(7):2474–2485

H. Liu et al.

Li G, Rabitz H, Hu J, Chen Z, Ju Y (2008) Regularized random-sampling high dimensional model representation (RS-HDMR). JMath Chem 43(3):1207–1232

Liu H, Xu S, Wang X (2015) Sequential sampling designs based onspace reduction. Eng Optim 47(7):867–884

Liu H, Xu S, Ma Y, Chen X, Wang X (2016a) An adaptive bayesiansequential sampling approach for global metamodeling. J MechDes 138(1):011–404

Liu H, Xu S, Wang X (2016b) Sampling strategies and metamodelingtechniques for engineering design: comparison and application. In:ASME Turbo Expo 2016: Turbomachinery Technical Conferenceand Exposition, ASME, pp V02CT45A019–V02CT45A019

Liu H, Xu S, Wang X, Meng J, Yang S (2016c) Optimal weightedpointwise ensemble of radial basis functions with different basisfunctions. AIAA J 54(10):3117–3133

Liu H, Ong YS, Cai J (2017a) An adaptive sampling approach forkriging metamodeling by maximizing expected prediction error .Comput Chem Eng 106:171–182

Liu H, Wang X, Xu S (2017b) Generalized radial basis function-basedhigh-dimensional model representation handling existing randomdata. J Mech Des 139(1):011–404

Liu Y, Hussaini MY, Okten G (2016d) Accurate construction of highdimensional model representation with applications to uncertaintyquantification. Reliab Eng Syst Saf 152:281–295

Morris MD, Mitchell TJ, Ylvisaker D (1993) Bayesian design andanalysis of computer experiments: use of derivatives in surfaceprediction. Technometrics 35(3):243–255

Mueller L, Alsalihi Z, Verstraete T (2013) Multidisciplinary optimiza-tion of a turbocharger radial turbine. J Turbomach 135(2):021–022

Rabitz H, Alis OF (1999) General foundations of high-dimensionalmodel representations. J Math Chem 25(2):197–233

Rabitz H, Alis OF, Shorter J, Shim K (1999) Efficient input-outputmodel representations. Comput Phys Commun 117(1-2):11–20

Razavi S, Tolson BA, Burn DH (2012) Review of surrogate modelingin water resources. Water Resour Res 48(7):1–32

Rippa S (1999) An algorithm for selecting a good value for the param-eter c in radial basis function interpolation. Adv Comput Math11(2):193–210

Shan S, Wang GG (2010a) Metamodeling for high dimensionalsimulation-based design problems. J Mech Des 132(5):051–009

Shan S, Wang GG (2010b) Survey of modeling and optimiza-tion strategies to solve high-dimensional design problems withcomputationally-expensive black-box functions. Struct Multidis-cip Optim 41(2):219–241

Shan S, Wang GG (2011) Turning black-box functions into whitefunctions. J Mech Des 133(3):031–003

Sobol IM (1993) Sensitivity estimates for nonlinear mathematicalmodels. Math Model Comput Exper 1(4):407–414

Sobol IM (2003) Theorems and examples on high dimensional modelrepresentation. Reliab Eng Syst Saf 79(2):187–193

Tang L, Wang H, Li G (2013) Advanced high strength steel spring-back optimization by projection-based heuristic global searchalgorithm. Mater Des 43:426–437

Tunga MA, Demiralp M (2005) A factorized high dimensional modelrepresentation on the nodes of a finite hyperprismatic regular grid.Appl Math Comput 164(3):865–883

Ulaganathan S, Couckuyt I, Dhaene T, Degroote J, Laermans E(2016) High dimensional kriging metamodelling utilising gradientinformation. Appl Math Model 40(9):5256–5270

Viana FA, Haftka RT, Steffen V (2009) Multiple surrogates: howcross-validation errors can help us to obtain the best predictor.Struct Multidiscip Optim 39(4):439–457

Wang GG, Shan S (2007) Review of metamodeling techniques in sup-port of engineering design optimization. J Mech Des 129(4):370–380

Wang SW, Georgopoulos PG, Li G, Rabitz H (2003) Randomsampling- high dimensional model representation (RS-HDMR)with nonuniformly distributed variables: Application to an inte-grated multimedia/multipathway exposure and dose model fortrichloroethylene. J Phys Chem A 107(23):4707–4716

Xu S, Liu H, Wang X, Jiang X (2014) A robust error-pursuingsequential sampling approach for global metamodeling based onvoronoi diagram and cross validation. J Mech Des 136(7):071–009

Yang Q, Xue D (2015) Comparative study on influencing factors inadaptive metamodeling. Eng Comput 31(3):561–577