Embed Size (px)
Citation preview
RESEARCH POSTER PRESENTATION DESIGN © 2012
www.PosterPresentations.com
(—THIS SIDEBAR DOES NOT PRINT—)
DES IGN GU IDE
This PowerPoint 2007 template produces a 36”x48” presentation
poster. You can use it to create your research poster and save
valuable time placing titles, subtitles, text, and graphics.
We provide a series of online tutorials that will guide you
through the poster design process and answer your poster
production questions. To view our template tutorials, go online
to PosterPresentations.com and click on HELP DESK.
When you are ready to print your poster, go online to
PosterPresentations.com
Need assistance? Call us at 1.510.649.3001
QU ICK START
Zoom in and out As you work on your poster zoom in and out to the level that is
more comfortable to you.
Go to VIEW > ZOOM.
Title, Authors, and Affiliations Start designing your poster by adding the title, the names of the authors,
and the affiliated institutions. You can type or paste text into the provided
boxes. The template will automatically adjust the size of your text to fit the
title box. You can manually override this feature and change the size of your
text.
TIP: The font size of your title should be bigger than your name(s) and
institution name(s).
Adding Logos / Seals Most often, logos are added on each side of the title. You can insert a logo
by dragging and dropping it from your desktop, copy and paste or by going
to INSERT > PICTURES. Logos taken from web sites are likely to be low
quality when printed. Zoom it at 100% to see what the logo will look like on
the final poster and make any necessary adjustments.
TIP: See if your school’s logo is available on our free poster templates
page.
Photographs / Graphics You can add images by dragging and dropping from your desktop, copy and
paste, or by going to INSERT > PICTURES. Resize images proportionally by
holding down the SHIFT key and dragging one of the corner handles. For a
professional-looking poster, do not distort your images by enlarging them
disproportionally.
Image Quality Check Zoom in and look at your images at 100% magnification. If they look good
they will print well.
ORIGINAL DISTORTED Corner handles
Go
od
pri
nti
ng
qu
alit
y
Bad
pri
nti
ng
qu
alit
y
QU ICK START ( con t . )
How to change the template color theme You can easily change the color theme of your poster by going to the DESIGN
menu, click on COLORS, and choose the color theme of your choice. You can
also create your own color theme.
You can also manually change the color of your background by going to VIEW
> SLIDE MASTER. After you finish working on the master be sure to go to
VIEW > NORMAL to continue working on your poster.
How to add Text The template comes with a number of pre-formatted
placeholders for headers and text blocks. You can
add more blocks by copying and pasting the existing
ones or by adding a text box from the HOME menu.
Text size Adjust the size of your text based on how much content you have to present.
The default template text offers a good starting point. Follow the
conference requirements.
How to add Tables To add a table from scratch go to the INSERT menu and
click on TABLE. A drop-down box will help you select rows and
columns.
You can also copy and a paste a table from Word or another PowerPoint
document. A pasted table may need to be re-formatted by RIGHT-CLICK >
FORMAT SHAPE, TEXT BOX, Margins.
Graphs / Charts You can simply copy and paste charts and graphs from Excel or Word. Some
reformatting may be required depending on how the original document has
been created.
How to change the column configuration RIGHT-CLICK on the poster background and select LAYOUT to see the column
options available for this template. The poster columns can also be
customized on the Master. VIEW > MASTER.
How to remove the info bars If you are working in PowerPoint for Windows and have finished your poster,
save as PDF and the bars will not be included. You can also delete them by
going to VIEW > MASTER. On the Mac adjust the Page-Setup to match the
Page-Setup in PowerPoint before you create a PDF. You can also delete them
from the Slide Master.
Save your work Save your template as a PowerPoint document. For printing, save as
PowerPoint of “Print-quality” PDF.
Print your poster When you are ready to have your poster printed go online to
PosterPresentations.com and click on the “Order Your Poster” button.
Choose the poster type the best suits your needs and submit your order. If
you submit a PowerPoint document you will be receiving a PDF proof for
your approval prior to printing. If your order is placed and paid for before
noon, Pacific, Monday through Friday, your order will ship out that same day.
Next day, Second day, Third day, and Free Ground services are offered. Go to
PosterPresentations.com for more information.
Student discounts are available on our Facebook page.
Go to PosterPresentations.com and click on the FB icon.
© 2013 PosterPresentations.com 2117 Fourth Street , Unit C Berkeley CA 94710
In spite of advances in computer capacity and speed, the
enormous computational cost of scientific and engineering
simulations and time constraints makes it impractical to rely
exclusively on simulation codes. A preferable strategy is to
replace the expensive simulations using surrogate models
Motivation
• Surrogate Modeling: Approximate a high dimensional function
defined on d-dimensional domain D
• Modeling and prediction using Random Forest regressor
• 25 realizations of {20,40,60,80,100} training samples
• Performance is measured based on 2500 testing samples
Poisson Disk Sampling (PDS)
Maximum sample size for a fixed disk radius:
Theoretical Bounds for PDS Empirical Analysis of Sampling Quality
1 Syracuse University, 2 Lawrence Livermore National Laboratory
Bhavya Kailkhura1,2, Jayaraman J. Thiagarajan2, Peer-Timo Bremer2, and Pramod K Varshney1
Theoretical Guarantees For Poisson Disk Sampling Using Pair Correlation Function
Importance of Experiment Design
Importance of uniformity to control aliasing
References
Maximum radius for a fixed sample size:
Challenge: There do not exist any methods to quantitatively
understand Poisson disk sampling
1. A. Lagae and P. Dutre, “A comparison of methods for generating poisson disk distributions,” Computer Graphics Forum, vol. 27, no. 1, pp. 114–129, 2008.
2. D. Heck, T. Schlomer, and O. Deussen, “Blue noise sampling with controlled aliasing,” ACM Trans. Graph., vol. 32, no. 3, pp. 25:1–25:12, Jul. 2013.
3. M. S. Ebeida, S. A. Mitchell, A. Patney, A. A. Davidson, and J. D. Owens, “A simple algorithm for maximal poisson disk sampling in high dimensions,” Computer Graphics Forum, vol. 31, no. 2pt4, pp. 785–794, 2012.
Conclusions
RESEARCH POSTER PRESENTATION DESIGN © 2012
www.PosterPresentations.com
(—THIS SIDEBAR DOES NOT PRINT—)
DES IGN GUIDE
This PowerPoint 2007 template produces a 36”x48” presentation
poster. You can use it to create your research poster and save
valuable time placing titles, subtitles, text, and graphics.
We provide a series of online tutorials that will guide you
through the poster design process and answer your poster
production questions. To view our template tutorials, go online
to PosterPresentations.com and click on HELP DESK.
When you are ready to print your poster, go online to
PosterPresentations.com
Need assistance? Call us at 1.510.649.3001
QUICK START
Zoom in and out As you work on your poster zoom in and out to the level that is more comfortable to you.
Go to VIEW > ZOOM.
Title, Authors, and Affiliations Start designing your poster by adding the title, the names of the authors,
and the affiliated institutions. You can type or paste text into the provided boxes. The template will automatically adjust the size of your text to fit the
title box. You can manually override this feature and change the size of your
text.
TIP: The font size of your title should be bigger than your name(s) and institution name(s).
Adding Logos / Seals Most often, logos are added on each side of the title. You can insert a logo by dragging and dropping it from your desktop, copy and paste or by going
to INSERT > PICTURES. Logos taken from web sites are likely to be low
quality when printed. Zoom it at 100% to see what the logo will look like on the final poster and make any necessary adjustments.
TIP: See if your school’s logo is available on our free poster templates
page.
Photographs / Graphics You can add images by dragging and dropping from your desktop, copy and
paste, or by going to INSERT > PICTURES. Resize images proportionally by
holding down the SHIFT key and dragging one of the corner handles. For a professional-looking poster, do not distort your images by enlarging them
disproportionally.
Image Quality Check Zoom in and look at your images at 100% magnification. If they look good they will print well.
ORIGINAL( DISTORTED(
Corner&handles&
Goo
d&prin/n
g&qu
ality
&
Bad&prin/n
g&qu
ality
&
QUICK START (cont . )
How to change the template color theme You can easily change the color theme of your poster by going to the DESIGN menu, click on COLORS, and choose the color theme of your choice. You can
also create your own color theme.
You can also manually change the color of your background by going to VIEW > SLIDE MASTER. After you finish working on the master be sure to go to
VIEW > NORMAL to continue working on your poster.
How to add Text The template comes with a number of pre-formatted
placeholders for headers and text blocks. You can add more blocks by copying and pasting the existing
ones or by adding a text box from the HOME menu.
Text size Adjust the size of your text based on how much content you have to present.
The default template text offers a good starting point. Follow the conference requirements.
How to add Tables To add a table from scratch go to the INSERT menu and click on TABLE. A drop-down box will help you select rows and
columns.
You can also copy and a paste a table from Word or another PowerPoint document. A pasted table may need to be re-formatted by RIGHT-CLICK >
FORMAT SHAPE, TEXT BOX, Margins.
Graphs / Charts You can simply copy and paste charts and graphs from Excel or Word. Some
reformatting may be required depending on how the original document has been created.
How to change the column configuration RIGHT-CLICK on the poster background and select LAYOUT to see the column options available for this template. The poster columns can also be
customized on the Master. VIEW > MASTER.
How to remove the info bars If you are working in PowerPoint for Windows and have finished your poster,
save as PDF and the bars will not be included. You can also delete them by going to VIEW > MASTER. On the Mac adjust the Page-Setup to match the
Page-Setup in PowerPoint before you create a PDF. You can also delete them from the Slide Master.
Save your work Save your template as a PowerPoint document. For printing, save as PowerPoint of “Print-quality” PDF.
Print your poster When you are ready to have your poster printed go online to PosterPresentations.com and click on the “Order Your Poster” button.
Choose the poster type the best suits your needs and submit your order. If
you submit a PowerPoint document you will be receiving a PDF proof for your approval prior to printing. If your order is placed and paid for before noon,
Pacific, Monday through Friday, your order will ship out that same day. Next day, Second day, Third day, and Free Ground services are offered. Go to
PosterPresentations.com for more information.
Student discounts are available on our Facebook page. Go to PosterPresentations.com and click on the FB icon.
©&2013&PosterPresenta/ ons.com&&&&&2117&Fourth&Street&,&Unit&C&&&&&&&&&&&&&&Berkeley&CA&94710&&&&&[email protected](
In spite of advances in computer capacity and speed, the
enormous computational cost of scientific and engineering
simulations (or experiments) makes it impractical to rely
exclusively on simulation codes. A preferable strategy is to
utilize surrogate modeling techniques, replacing the
expensive simulation model.
Mo=va=on(
Regression(Experiment(
Sampling Schemes:
25 realization of {20,40,60,80,100} training samples of the
following sampling techniques:
Regression(Experiment(.(.(.(
Performance Metric:
Performance of the regressor is measured based on 2500 testing
samples using following metrics
Results( Results(.(.(.(
Booth Function:
Ongoing(Research(
1&Syracuse&University,&2&Lawrence&Livermore&Na/ onal&Laboratory!
Bhavya&Kailkhura1,2,&Jayaraman&J.&Thiagarajan2,&and&PeerQTimo&Bremer2&
Does(sampling(scheme(impact(the(performance(of(Surrogate(Modeling?(
Main(Contribu=ons(Test Functions:
Different classes of benchmark Functions
Analysis and synthesis of sampling schemes
Ackley’s Function: Easom Function:
References(
Eggholder Function:
Surrogate Model (modeling and predication)
Review and Release #
!!Surrogate!Modeling!!
!!!!!Design!Variables!
)temperature,)angle,))
))length,)frequency).).).)
))))
!!!!!Response!Variables!
))volume,)pressure,)stress).).).)
Surrogate!Model!Cheap!
Prototyping) Visualiza&on) Op&miza&on) ))))))CAD)
Design!of!Experiments!(DOE)!
Design)Space)
1
0.8
0.6
0.4
0.2
00
0.2
0.4
0.6
0.8
22
10
12
14
24
16
18
20
1
1
0.8
0.6
0.4
0.2
00
0.2
0.4
0.6
0.8
0
15
10
5
20
25
1
Surrogate!model!using!Kriging!(built!using!20!samples)!
Ackley’s)func&on)
Design!of!Experiments!(DOE)!
Design)Space)
1
0.8
0.6
0.4
0.2
00
0.2
0.4
0.6
0.8
22
10
12
14
24
16
18
20
1
1
0.8
0.6
0.4
0.2
00
0.2
0.4
0.6
0.8
0
15
10
5
20
25
1
Surrogate!model!using!Kriging!(built!using!20!samples)!
Ackley’s)func&on)
Automo&ve) Chemistry) Aerodynamics) Metallurgy) !!!!.!.!.!.!!
!!!!!Design!Variables!
)temperature,)angle,))))length,)frequency).).).)
))))
or) ) ) ) )))))real)world)experiment)
Simula1on!Model!!!!!!
Response!Variables!)
)volume,)pressure,)stress).).).)
Costly!
Automo&ve) Chemistry) Aerodynamics) Metallurgy) !!!!.!.!.!.!!
!!!!!Design!Variables!
)temperature,)angle,))))length,)frequency).).).)
))))
or) ) ) ) )))))real)world)experiment)
Simula1on!Model!!!!!!
Response!Variables!)
)volume,)pressure,)stress).).).)
Costly!
Costly!
An immediate question that a designer may have is based on
what criterion the various techniques should be used?
Existing literature lacks methods or procedures for testing
the relative merits of different methods in a useful manner.
First positive result on the topic:
• Methods for systematic and comprehensive comparative
studies of the various techniques.
• Inverse statistical mechanics technique for generating
superior sampling schemes.
• Recommendations for the appropriate use of surrogate
techniques and how common pitfalls can be avoided.
Kriging regressor:
Z(x) is stochastic process with correlation function
Two different correlation functions are chosen
• Gaussian with polynomial of order 2
• Exponential with polynomial of order 2
2 American Institute of Aeronautics and Astronautics
Regression, ANN, and the Moving Least Square—are
compared for the construction of safety related
functions in automotive crash analysis, for a relative
small sampling size. Simpson (1999) presents the
results of ongoing work investigating different
metamodeling techniques—response surfaces, kriging models, radial basis functions, and MARS models—on
a variety of engineering test problems. Although the
existing studies provide useful insights into the various
approaches considered, a common limitation is that the
tests are restricted to a very small group of methods and
test problems, and in many cases only one problem due
to the expenses associated with testing. Moreover,
when using multiple test problems, it is often difficult
to make comparisons between the test problems when
they belong to different classes of problems (e.g.,
linear, quadratic, nonlinear, etc.).
It is our belief that various factors contribute to the success of a given metamodeling technique, ranging
from the nonlinearity of the model behavior, to the
dimensionality and data sampling technique, to the
internal parameter settings of the various techniques.
We contend that instead of using accuracy as the only
measure, multiple metrics for comparison should be
considered based on multiple modeling criteria, such as
accuracy, efficiency, robustness, model transparency,
and simplicity. Overall, the knowledge of the
performance of different metamodeling techniques with
respect to different modeling criteria is of utmost importance to designers when trying to choose an
appropriate technique for a particular application.
In this work, we present preliminary results from a
systematic comparative study which provides insightful
observations into the performance of various
metamodeling techniques under different modeling
criteria, and the impact of the contributing factors to
their success. A set of mathematical and engineering
problems has been selected to represent different
classes of problems with different degrees of
nonlinearity, different dimensions, and noisy versus
smooth behaviors. Relative large, small, and scarce sample sets are also used for each test problem. Four
promising metamodeling techniques, namely,
Polynomial Regression (PR), Kriging (KG),
Multivariate Adaptive Regression Splines (MARS), and
Radial Basis Functions (RBF), are compared in this
study. Although ANN is a well-known technique, it is
not included in our study due to the large amount of
trail-and-error associated with the use of this technique.
2 Metamodeling Techniques The principle features of the four metamodeling
techniques compared in our study are described in the
following sections.
2.1 Polynomial Regression (PR)
PR models have been applied by a number of
researchers (Engelund, et al, 1993; Unal, et al., 1996;
Vitali, et al., 1997; Venkataraman, et al., 1997; Venter,
et al., 1997; Chen, et al., 1996; Simpson, et al., 1997) in
designing complex engineering systems. A second-order polynomial model can be expressed as:
å å å å+++== =
k
i
k
i i jjiijiiiiio xxxxy
1 1
2ˆ bbbb (1)
When creating PR models, it is possible to identify
the significance of different design factors directly from
the coefficients in the normalized regression model.
For problems with a large dimension, it is important to
use linear or second-order polynomial models to narrow
the design variables to the most critical ones. In
optimization, the smoothing capability of polynomial regression allows quick convergence of noisy functions
(see, e.g., Guinta, et al., 1994). In spite of the
advantages, there is always a drawback when applying
PR to model highly nonlinear behaviors. Higher-order
polynomials can be used; however, instabilities may
arise (Barton, 1992), or it may be too difficult to take
sufficient sample data to estimate all of the coefficients
in the polynomial equation, particularly in large
dimensions. In this work, linear and second-order PR
models are considered.
2.2 Kriging Method (KG) A kriging model postulates a combination of a
polynomial model and departures of the form:
å +==
k
jjj xZxfy
1
)()(ˆ b , (2)
where Z(x) is assumed to be a realization of a
stochastic process with mean zero and spatial
correlation function given by
Cov[Z(xi),Z(xj)] = s2 R(xi, xj), (3)
where s2 is the process variance and R is the
correlation. A variety of correlation functions can be
chosen (cf., Simpson, et al., 1998); however, the
Gaussian correlation function proposed in (Sacks, et al.,
1989) is the most frequently used. Furthermore, fj(x) in
Eqn. 2 is typically taken as a constant term. In our
study, we use a constant term for fj(x) and a Gaussian
correlation function with p=2 and k q parameters, one q for each of the k dimensions in the design space.
In addition to being extremely flexible due to the wide range of the correlation functions, the kriging
method has advantages in that it provides a basis for a
stepwise algorithm to determine the important factors,
and the same data can be used for screening and
building the predictor model (Welch, et al., 1992). The
major disadvantage of the kriging process is that model
construction can be very time-consuming. Determining
the maximum likelihood estimates of the q parameters used to fit the model is a k-dimensional optimization
2 American Institute of Aeronautics and Astronautics
Regression, ANN, and the Moving Least Square—are
compared for the construction of safety related
functions in automotive crash analysis, for a relative
small sampling size. Simpson (1999) presents the
results of ongoing work investigating different
metamodeling techniques—response surfaces, kriging models, radial basis functions, and MARS models—on
a variety of engineering test problems. Although the
existing studies provide useful insights into the various
approaches considered, a common limitation is that the
tests are restricted to a very small group of methods and
test problems, and in many cases only one problem due
to the expenses associated with testing. Moreover,
when using multiple test problems, it is often difficult
to make comparisons between the test problems when
they belong to different classes of problems (e.g.,
linear, quadratic, nonlinear, etc.).
It is our belief that various factors contribute to the success of a given metamodeling technique, ranging
from the nonlinearity of the model behavior, to the
dimensionality and data sampling technique, to the
internal parameter settings of the various techniques.
We contend that instead of using accuracy as the only
measure, multiple metrics for comparison should be
considered based on multiple modeling criteria, such as
accuracy, efficiency, robustness, model transparency,
and simplicity. Overall, the knowledge of the
performance of different metamodeling techniques with
respect to different modeling criteria is of utmost importance to designers when trying to choose an
appropriate technique for a particular application.
In this work, we present preliminary results from a
systematic comparative study which provides insightful
observations into the performance of various
metamodeling techniques under different modeling
criteria, and the impact of the contributing factors to
their success. A set of mathematical and engineering
problems has been selected to represent different
classes of problems with different degrees of
nonlinearity, different dimensions, and noisy versus
smooth behaviors. Relative large, small, and scarce sample sets are also used for each test problem. Four
promising metamodeling techniques, namely,
Polynomial Regression (PR), Kriging (KG),
Multivariate Adaptive Regression Splines (MARS), and
Radial Basis Functions (RBF), are compared in this
study. Although ANN is a well-known technique, it is
not included in our study due to the large amount of
trail-and-error associated with the use of this technique.
2 Metamodeling Techniques The principle features of the four metamodeling
techniques compared in our study are described in the
following sections.
2.1 Polynomial Regression (PR)
PR models have been applied by a number of
researchers (Engelund, et al, 1993; Unal, et al., 1996;
Vitali, et al., 1997; Venkataraman, et al., 1997; Venter,
et al., 1997; Chen, et al., 1996; Simpson, et al., 1997) in
designing complex engineering systems. A second-order polynomial model can be expressed as:
å å å å+++== =
k
i
k
i i jjiijiiiiio xxxxy
1 1
2ˆ bbbb (1)
When creating PR models, it is possible to identify
the significance of different design factors directly from
the coefficients in the normalized regression model.
For problems with a large dimension, it is important to
use linear or second-order polynomial models to narrow
the design variables to the most critical ones. In
optimization, the smoothing capability of polynomial regression allows quick convergence of noisy functions
(see, e.g., Guinta, et al., 1994). In spite of the
advantages, there is always a drawback when applying
PR to model highly nonlinear behaviors. Higher-order
polynomials can be used; however, instabilities may
arise (Barton, 1992), or it may be too difficult to take
sufficient sample data to estimate all of the coefficients
in the polynomial equation, particularly in large
dimensions. In this work, linear and second-order PR
models are considered.
2.2 Kriging Method (KG) A kriging model postulates a combination of a
polynomial model and departures of the form:
å +==
k
jjj xZxfy
1
)()(ˆ b , (2)
where Z(x) is assumed to be a realization of a
stochastic process with mean zero and spatial
correlation function given by
Cov[Z(xi),Z(xj)] = s2 R(xi, xj), (3)
where s2 is the process variance and R is the correlation. A variety of correlation functions can be
chosen (cf., Simpson, et al., 1998); however, the
Gaussian correlation function proposed in (Sacks, et al.,
1989) is the most frequently used. Furthermore, fj(x) in
Eqn. 2 is typically taken as a constant term. In our
study, we use a constant term for fj(x) and a Gaussian
correlation function with p=2 and k q parameters, one q for each of the k dimensions in the design space.
In addition to being extremely flexible due to the wide range of the correlation functions, the kriging
method has advantages in that it provides a basis for a
stepwise algorithm to determine the important factors,
and the same data can be used for screening and
building the predictor model (Welch, et al., 1992). The
major disadvantage of the kriging process is that model
construction can be very time-consuming. Determining
the maximum likelihood estimates of the q parameters used to fit the model is a k-dimensional optimization
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Random Latent Hyper Cube Step Blue Noise Poisson Disk
Challenge: How to categorize various sampling schemes?
Ackley’s Eggholder Booth Easom
Challenge: How to classify or compare various functions?
• Root mean square error (RMSE): overall accuracy
• Maximum absolute error (MAX): local accuracy
• R squared value: goodness of fit
13
leave-one-out cross-validation is actually a measurement for degrees of insensitivity of a
metamodel to lost information at its data points, while an insensitive metamodel is not
necessarily accurate. A “validated” model by leave-one-out could be far from the actual as the
data points may not be able to capture the actual. Designers are in danger of accepting an
inaccurate metamodel that is insensitive to lost information at data points, and inaccurate and
insensitive metamodels might be the results of poor experimental designs (clustering points or
correlated data points). On the other hand, with leave-one-out cross validation we are in danger
of rejecting an accurate metamodel that is also sensitive to lost information at data points.
Given that cross validation is insufficient for assessing models, employing additional points is
essential in metamodel validation. When additional points are used for validation, there are a
number of different measures of model accuracy. The first two are the root mean square error
(RMSE) and the maximum absolute error (MAX), defined below:
m
yy
RMSE
m
i
ii∑=
-
= 1
2)ˆ(
(1)
miyyMAX ii ,,1|,ˆ|max L=-= (2)
where m is the number of validation points; iy is the predicted value for the observed value yi.
The lower the value of RMSE and/or MAX, the more accurate the metamodel. RMSE is used to
gauge the overall accuracy of the model, while MAX is used to gauge the local accuracy of the
model. An additional measure is also used is the R square value, i.e.,
∑
∑
=
=
-
-
-=m
i
i
m
i
ii
yy
yy
R
1
2
1
2
2
)(
)ˆ(
1 (3)
13
leave-one-out cross-validation is actually a measurement for degrees of insensitivity of a
metamodel to lost information at its data points, while an insensitive metamodel is not
necessarily accurate. A “validated” model by leave-one-out could be far from the actual as the
data points may not be able to capture the actual. Designers are in danger of accepting an
inaccurate metamodel that is insensitive to lost information at data points, and inaccurate and
insensitive metamodels might be the results of poor experimental designs (clustering points or
correlated data points). On the other hand, with leave-one-out cross validation we are in danger
of rejecting an accurate metamodel that is also sensitive to lost information at data points.
Given that cross validation is insufficient for assessing models, employing additional points is
essential in metamodel validation. When additional points are used for validation, there are a
number of different measures of model accuracy. The first two are the root mean square error
(RMSE) and the maximum absolute error (MAX), defined below:
m
yy
RMSE
m
i
ii∑=
-
= 1
2)ˆ(
(1)
miyyMAX ii ,,1|,ˆ|max L=-= (2)
where m is the number of validation points; iy is the predicted value for the observed value yi.
The lower the value of RMSE and/or MAX, the more accurate the metamodel. RMSE is used to
gauge the overall accuracy of the model, while MAX is used to gauge the local accuracy of the
model. An additional measure is also used is the R square value, i.e.,
∑
∑
=
=
-
-
-=m
i
i
m
i
ii
yy
yy
R
1
2
1
2
2
)(
)ˆ(
1 (3)
13
leave-one-out cross-validation is actually a measurement for degrees of insensitivity of a
metamodel to lost information at its data points, while an insensitive metamodel is not
necessarily accurate. A “validated” model by leave-one-out could be far from the actual as the
data points may not be able to capture the actual. Designers are in danger of accepting an
inaccurate metamodel that is insensitive to lost information at data points, and inaccurate and
insensitive metamodels might be the results of poor experimental designs (clustering points or
correlated data points). On the other hand, with leave-one-out cross validation we are in danger
of rejecting an accurate metamodel that is also sensitive to lost information at data points.
Given that cross validation is insufficient for assessing models, employing additional points is
essential in metamodel validation. When additional points are used for validation, there are a
number of different measures of model accuracy. The first two are the root mean square error
(RMSE) and the maximum absolute error (MAX), defined below:
m
yy
RMSE
m
i
ii∑=
-
= 1
2)ˆ(
(1)
miyyMAX ii ,,1|,ˆ|max L=-= (2)
where m is the number of validation points; iy is the predicted value for the observed value yi.
The lower the value of RMSE and/or MAX, the more accurate the metamodel. RMSE is used to
gauge the overall accuracy of the model, while MAX is used to gauge the local accuracy of the
model. An additional measure is also used is the R square value, i.e.,
∑
∑
=
=
-
-
-=m
i
i
m
i
ii
yy
yy
R
1
2
1
2
2
)(
)ˆ(
1 (3)
In statistical mechanics, the Pair Correlation Function (PCF)
describes how density varies as a function of distance.
0 0.05 0.1 0.15 0.2 0.250
10
20
30
40
50
60
0 0.05 0.1 0.15 0.2 0.250
2
4
6
8
10
12
0 0.05 0.1 0.15 0.2 0.250
0.2
0.4
0.6
0.8
1
1.2
0 0.05 0.1 0.15 0.2 0.250
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Random Latent Hyper Cube Step Blue Noise Poisson Disk
PCF (g(r)) is related to power spectral density (PSD) via
Hankel Transform. Thus gives control over spectral properties
of sampling pattern for synthesis.
Inverse statistical mechanics technique for sampling
Weighted least square based PCF matching problem:
min ||g(r)-gt(r)||
Fast gradient descent algorithm to solve the problem.
Power spectral density answers the question “which
frequency contains the signal’s power?”
P(v)=|abs(F(signal))|2 where F(.) is fourier transform
5 10 15 20 25 30 35 40 45 50
5
10
15
20
25
30
35
40
45
50
100
200
300
400
500
600
700
800
900
1000
5 10 15 20 25 30 35 40 45 50
5
10
15
20
25
30
35
40
45
50
5 10 15 20 25 30 35 40 45 50
5
10
15
20
25
30
35
40
45
50700
800
900
1000
1100
1200
1300
5 10 15 20 25 30 35 40 45 50
5
10
15
20
25
30
35
40
45
501.798
1.799
1.8
1.801
1.802
1.803
1.804
1.805
Ackley’s Eggholder Booth Easom
number of samples (x 20)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
RM
SE
0.8
1
1.2
1.4
1.6
1.8
2
2.2
2.4
2.6
2.8
LHS
Random
Poisson disk
Step blue noise
number of samples (x 20)0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
MA
X
2
4
6
8
10
12
14
16
LHS
Random
Poisson disk
Step blue noise
number of samples (x 20)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
R s
quar
e
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
LHS
Random
Poisson disk
Step blue noise
Gaussian&correla/ on&func/ on&
number of samples (x 20)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
RM
SE
0.5
1
1.5
2
2.5
3
3.5
LHS
Random
Poisson disk
Step blue noise
number of samples (x 20)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
MA
X
4
6
8
10
12
14
16
18
LHS
Random
Poisson disk
Step blue noise
number of samples (x 20)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
R s
quar
e
-2
-1.5
-1
-0.5
0
0.5
1
1.5
LHS
Random
Poisson disk
Step blue noise
Exponen/ al&correla/ on&func/ on&
number of samples (x 20)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
RM
SE
250
300
350
400
450
500
LHS
Random
Poisson disk
Step blue noise
number of samples (x 20)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
MA
X
800
1000
1200
1400
1600
1800
2000
2200
LHS
Random
Poisson disk
Step blue noise
number of samples (x 20)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
R s
quar
e
-2
-1.5
-1
-0.5
0
0.5
LHS
Random
Poisson disk
Step blue noise
Gaussian&correla/ on&func/ on&
number of samples (x 20)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
RM
SE
250
300
350
400
450
500
LHS
Random
Poisson disk
Step blue noise
number of samples (x 20)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5M
AX
800
1000
1200
1400
1600
1800
2000
2200
LHS
Random
Poisson disk
Step blue noise
number of samples (x 20)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
R s
quar
e
-2
-1.5
-1
-0.5
0
0.5
LHS
Random
Poisson disk
Step blue noise
Exponen/ al&correla/ on&func/ on&
number of samples (x 20)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
RM
SE
#10-13
1.2
1.4
1.6
1.8
2
2.2
2.4
2.6
LHS
Random
Poisson disk
Step blue noise
number of samples (x 20)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
MA
X
# 10-12
0.7
0.8
0.9
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
LHS
Random
Poisson disk
Step blue noise
number of samples (x 20)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
R s
quar
e
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Gaussian&correla/ on&func/ on&
number of samples (x 20)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
RM
SE
#10-12
0
0.2
0.4
0.6
0.8
1
1.2
LHS
Random
Poisson disk
Step blue noise
number of samples (x 20)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
MA
X
#10-12
0
0.2
0.4
0.6
0.8
1
1.2
1.4
LHS
Random
Poisson disk
Step blue noise
number of samples (x 20)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
R s
quar
e
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Exponen/ al&correla/ on&func/ on&
number of samples (x 20)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
RM
SE
# 10-3
-6
-4
-2
0
2
4
6
8
Poisson disk
Step blue noise
Random
LHS
number of samples (x 20)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
MA
X
-0.04
-0.02
0
0.02
0.04
0.06
0.08
Poisson disk
Step blue noise
Random
LHS
number of samples (x 20)
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
R s
qua
re
-1500
-1000
-500
0
500
1000
Poisson disk
Step blue noise
Random
LHS
Gaussian&correla/ on&func/ on&
• Simpson,&T.W.,&Peplinski,&J.,&Koch,&P.N.,&and&Allen,&J.K.,&&"Metamodels&for&ComputerQBased&Engineering&Design:&Survey&and&Recommenda/ ons,”&Engineering&with&Computers,&Vol.&17,&No.&2,&2001,&pp.&129Q150.&
• Yesilyurt,&S.&and&Patera,&A.T.&(1995)&"Surrogates&for&Numerical&Simula/ ons:&Op/miza/ on&of&EddyQPromoter&Heat&Exchangers."&Computer&Methods&in&Applied&Mechanics&and&Engineering,&21(1Q4),&231Q257.&
Recommenda=ons(
• Experiments with several other surrogate modes
• Establishing bounds on regression error
• Pair Correlation analysis allows theoretical analysis of
Poisson disk sampling
• Upper bounds on sample size enable early termination of
existing Maximal PDS algorithms
• Effect of the choice of disk radius on the resulting
spectrum reveals trade-off between low frequency
aliasing and high frequency oscillations
Ackley Eggholder Booth
Uniform
Min Distance
r rmin
Maximal PDS: A PDS is maximal if no more
points can be inserted
PDS using Pair Correlation Function (PCF)
In statistical mechanics, PCF describes how density varies as a
function of distance from a reference. PCF, G(r), is related to
power spectral density (PSD) via the Hankel Transform
Using the PCF-PSD relation, we can derive the radially averaged
PSD of Poisson disk sampling:
Necessary Conditions
PSD of PDS
Existing approaches for MPDS experimentally
obtain the bounds for N
Desired Properties:
• Zero for low frequencies which indicates the range of
frequencies that can be represented with almost no aliasing
• Constant for high frequencies which reduces the risk of
aliasing
Impact of the choice of disk radius
Increased peak height results in
higher low frequency aliasing
and larger high frequency
oscillations
Trade-off :
• Spectrum tends to be close to
zero at low frequencies
• Significant increase in
oscillations at high frequencies
Random Latin Hyper Cube Poisson Disk Sampling
PAIR CORRELATION RADIALLY AVERAGED PSD
Validation of Poisson Disk Sample Designs
N = 3000 N = 5000 N = 7000
Close to theoretical
bound
State-of-the-art
MPDS
