Applied Mathematical Sciences, Vol. 10, 2016, no. 20, 989 - 1011 HIKARI Ltd, www.m-hikari.com

http://dx.doi.org/10.12988/ams.2016.511712

Attraction Force Optimization (AFO): A Deterministic

Nature-Inspired Heuristic for Solving Optimization

Problems in Stochastic Simulation

I. Bendato 1, L. Cassettari 1, P. G. Giribone 2 and S. Fioribello 1

1DIME: Department of Mechanical Engineering, Energetics

Management and Transports University of Genoa, Italy

2CARIGE Bank - Financial Administration (Engineering and Pricing)

Genoa, Italy Copyright © 2015 I. Bendato et. al. This article is distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

The paper presents a new optimization heuristic called AFO - Attraction Force Optimization, able to maximize discontinuous, non-differentiable and highly non-linear functions in discrete simulation problems. The algorithm was developed specifically to overcome the limitations of traditional search algorithms in optimization problems performed on discrete-event simulation models used, for example, to study industrial systems and processes. Such applications are characterized by three particular aspects: the response surfaces of the objective function is not known to the experimenter, a few number of independent variables are involved, very high computational time for each single simulation experiment. In this context it is therefore essential to use an optimization algorithm that on one hand tries to explore as effectively as possible the entire domain of investigation but, in the same time, does not require an excessive number of experiments. The article, after a quick overview of the most known optimization techniques, explains the properties of AFO, its strengths and limitations compared to other search algorithms. The operating principle of the heuristic, inspired by the laws of attraction occurring in nature, is discussed in detail in the case of 1, 2 and N-dimensional functions from a theoretical and applicative point of view. The algorithm was then validated using the most common 2-dimensional and N-dimensio-

990 I. Bendato et al. nal benchmark functions. The results are absolutely positive if compared, for the same initial conditions, with the traditional methods up to 10-dimensional vector spaces. A higher number of independent variables is generally not of interest for discrete simulation optimization problems in industrial applications (our research field). Keywords: Stochastic Simulation; Nature-Inspired Heuristic; Mathematical Computing; Optimization 1. Introduction The term "optimization" refers to the process of searching the minimum or the maximum of an objective function (target function). The optimization problems can be complex as to be virtually impossible to solve them by an analytic way. The complexity is determined primarily by the number of variables that define the problem and, often, by the presence of some constraints. The analytical solution is possible only if few variables are involved and when the target functions are rather simple and smooth. In practice, to solve an optimization problem, it is possible to use a set of numerical procedures that can be classified relatively to the mathematical logic that characterizes their behavior: • Gradient-Based methods: algorithms that base their research on the calculation of the partial derivatives. The most popular are Steepest Descent, Newton’s method and quasi-Newton methods. • Direct Search methods: are best known as unconstrained optimization techniques that do not explicitly use derivatives but search the optimum point by calculating only the images of the target function (function evaluations). This class includes the Simplex Method, the Nelder-Mead Simplex Algorithm and the Pattern Search techniques. • Nature-Inspired methods: algorithms whose functioning simulates a social behavior or a physical phenomenon present in nature. This category includes Genetic Algorithms, Annealing Search, Particle Swarm, Ant colony, Imperialistic competitive, Artificial Bee Colony, Harmony search and Fire Flights. 1.1. Gradient-Based methods As mentioned previously, the Gradient-Based algorithms are techniques that use information resulting from the calculation of the partial derivatives of the target function [8-10]. The Steepest Descent is a first-order optimization algorithm that, in order to determine a point of local minimum of the target function, uses the information from the calculation of the gradient of the function [2, 26, 28]. It gets the direction of maximum descent (hence the name of the method) using a recursive estimation of the gradient at the current point )( nxf∇ [11, 27].

Attraction force optimization 991 The Newton’s method to find the optimum point, uses not only the determination of the gradient of the target function, but also the calculation of the Hessian matrix. Being a methodology of the second order, it proceeds by calculating both the gradient )(xf∇ and the Hessian matrix )( nxHf . To find the local minimum, the algorithm iteratively approximates the neighborhood function with a quadratic form and finds its minimum value. In general, the computational time of Newton's method is very high because at each iteration, in addition to having to calculate the Hessian matrix, the algorithm must also determine its inverse

)(1nxHf − . It is also well known that the Newton method is characterized by a

good convergence speed in the local optimum [15]. The Quasi-Newton (QN) method has the great advantage, compared to the Newton’s method, of not requiring the explicit computation of the Hessian matrix at each iteration; the latter is in fact determined by approximating the values of the gradient in the previous iterations [24]. Just according to the rule of calculation of

)( nxH , it is possible to distinguish different methodologies; between all the Quasi-Newtonian techniques cited in the literature, the Limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) is today the one that allow to more efficiently solve local optimization problems. The L-BFGS method iteratively finds a minimizer by approximating the inverse hessian matrix by information from last m iterations [12]. This innovation saves the memory storage and computational time drastically for large-scaled problems. 1.2. Direct Search Methods The Simplex Method developed by Spendley, Hext and Himsworth in 1962 basically relies in 2ℜ on triangulation operations and subsequent rollovers [22, 23]. Starting from a regular simplex (which in 2ℜ is configured as an equilateral triangle), the algorithm calculates the images of the function at the three vertices, identifying the "worst", ie the one that has the highest value when considering a minimization problem. It then made an overturn (reflection) of the vertex so identified with respect to the centroid of the remaining vertices: this will be the new vertex of the next iteration. The procedure comes to convergence when the vertex of the simplex having the best value of the objective function remains unchanged for k iterations [32]. The Nelder-Mead Simplex Algorithm represents a sort of extension of the algorithm of Spendley: similarly to the methodology described previously, it always works with a simplex, in n variables, always characterized by n +1 vertices. The advantage is the possibility of performing operations on the convex shape thus allowing to vary both the shape and the size: Reflection – Expansion – Contraction - Shrinkage [21]. In literature it is possible to find several applications to the industrial sector and, in particular, in minimization/maximization of objective function derived by Discrete Event Simulators [3, 19, 20].

992 I. Bendato et al. Pattern Search Methods are part of a family of numerical optimization algorithms that do not follow the classical procedures of the mathematical analysis. For this reason, these methods can be employed to determine the optimal point of functions that are not continuous or differentiable. This type of algorithm is particularly suitable in the case where the surface to be optimized presents strong discontinuities and / or points of non-differentiability. Reasoning in view of minimization, the technique works by identifying a search direction from the set of points in the neighborhood of the current point, selecting the one whose image may be smaller than the value of the current point [11, 31]. Among all the methodologies Pattern Search the Authors cite the deterministic ones, known in the literature as Generalized Pattern Search (GPS), which calculate a sequence of points of progressive approach to the optimum region [1]. 1.3 Nature Inspired Methods The Genetic Algorithms (GA) represent an optimization technique that is based on the concept of biological natural selection of the species [16]. These methods iteratively and dynamically modify a population of individuals: at each step, the algorithm randomly selects pairs from the current population, as candidates to be parents to produce children in the next generation. Through a series of successive generations, the population evolves toward an optimal solution. You can apply these methods to solve a particular range of optimization problems, which do not find efficient solutions using standard types of maximization algorithms: it makes particular reference to applications involving objective functions discontinuous, non-differentiable, stochastic, or highly nonlinear. The Simulated Annealing (SA) is a research methodology applicable to any non-convex optimization problem and bases its foundations in the simulation of particular aspects of thermodynamics; it was originally developed by Kirkpatrick in 1983 to solve combinatorial and discrete optimization problems. The SA was created as a method of simulation of quenching (annealing) of solids. The annealing is a process by which a solid, resulted in the molten state by heating, is then reported back to the solid state with the crystalline structure, controlling and reducing the temperature gradually until it reaches a minimum energy configuration particle [17]. The nature-inspired heuristic algorithms have proven to be more suitable to deal with highly nonlinear problems [7, 35, 36]. However, these techniques are characterized by a high number of function evaluations: a characteristic common to all procedures based on agents. Despite the wide range of consolidated methodologies the problem is constantly evolving and new contributions are added to those already existing in an attempt to overcome the inevitable limitations that each algorithm presents [13, 18, 25, 30, 33, 34, 37, 38].

Attraction force optimization 993 2. The Attraction Force Optimization (AFO) heuristic The new optimization technique "Attraction Force Optimization" (AFO) is a maximization heuristic inspired by the attractive forces present in nature, such as gravity or electricity. This algorithm tries to combine the strengths of the classical deterministic methods and the advantages of heuristics inspired by natural processes. Consequently AFO is characterized, on the one hand, by a greater analytical tractability, by tracking the convergence to the solution, the ability to always get the same result starting from the same initial conditions, and lastly it does not require an excessive number of function evaluations. On the other hand, in line with the modern philosophy, AFO is able to present a thorough exploration of the experimental domain, the use of rather simple calculation operators and, also, the presence of a population of points interacting between them. As mentioned above, the algorithm developed by the Authors is inspired by the physic problem of universal gravitation force [14]. The Newton's law of universal gravitation states that any two bodies of masses

1m and 2m in the universe attract each other with a force that is directly proportional to the product of their masses and inversely proportional to the square of the distance r between them:

221

rmmGF ⋅

= where:

2

2111067.6

kgNmG −⋅= .

Referring to the case of the Earth and the Moon, suppose you want to determine the equilibrium point ( MEQUILIBRIUx ) at which the attractive forces cancel each other. Defining x the distance between the Earth and the point of equilibrium and, consequently, by calling xr − the distance between this latter point and the Moon, to determine the coordinates in which the two forces are "offset" it follows:

( )22 xr

mmG

xmm

G MOONEQEQEARTH

−

⋅=

⋅

from which it is possible to obtain the quadratic equation in x that allows to find the equilibrium point:

EARTHMOON mmrx

/121±

=−

If you replace the values kgmEARTH

241097.5 ⋅= , kgmMOON24100735.0 ⋅= and

then mr 81084.3 ⋅= you actually get the numerical value of Mmx MEQUILIBRIU 346= .

994 I. Bendato et al. Consequently the point between the Earth and the Moon where the two forces cancel each other is 9/10 of the Earth-Moon distance. It is now described the analogy between the example introduced above and the heuristic proposed by the Authors. As it is known, the goal of optimization is to determine: max )(xf

Nx ℜ∈

where: UL xxx

≤≤ employing a number as much as possible reduced of iterations. Each instance of the decision variable x investigated is associated with an apparent mass defined "charge". The attractive force )(xf associated with each charge is proportional to the potential assumed by the vector field (the fitness function to be maximized) at that point. The number of charges considered at each iteration of the algorithm (experiment) represents the experimental population. The number of charges constituting the population, constant over time, is arranged according to a closed pattern having directions equipollent to the orthonormal basis generating the domain of investigation. 2.1 AFO characterization in 1-dimensional problems Let consider the decision variable x defined between a lower bound Lx and an upper bound Ux . The initialization of the algorithm is carried out by putting a charge on each of the two vertices of the segment between Lx and Ux . The initial schema S is completed placing between Lx and Ux , at the discretion of the experimenter, other m charges equidistant from each other, with 0≥m . The number m of charges to be included in the initial scheme, in addition to those "mandatory" placed at the vertices (that define the research domain) is completely arbitrary: the more charges are placed, the more accurate the algorithm is. Then the potential associated with each charge is assumed equal to the image of the objective function )(xf . The charge that in the initial scheme has the highest image (the greater value of potential associated), is defined basis charge and its coordinate is saved in an array. Once the basis charge ( BASISx ) has been identified, it is possible to relate it with each of the other charges in the scheme S by applying:

( )( )BASIS

i

CHARGEBASISiEQ

xfxf

rx i

+= −

1,

Attraction force optimization 995 where

iCHARGEBASISr − is the distance between the basis charge in the schema and the i-th charge. For each of the equilibrium points thus determined, their potential is estimated by calculating )( ,iEQxf . The current iteration (experiment) ends by selecting the basis charge and the equilibrium point farther from the basis charge. The basis charge and the equilibrium point become the new boundaries of the experimental domain, i.e. they are the generators for the new schema. The predetermined criteria used to stop the algorithms’ search are: • if the size of the schema (the length of the segment for N = 1) is below a defined tolerance (ε ) • if the maximum number of planned experiments (iterations) exceed a pre-assigned budget • if the maximum number of function evaluations has been reached.

FIG. 1 - Logical flow of AFO iterations for one-dimensional problems Once it has been defined the schema for the new experiment, it is possible to determine the basis charge for the new experiment. The algorithm, so set, could in some cases fall into the problem of a too rapid convergence. This could occur, for example, in the case where, in the initial schema, the basis charge has a potential far greater than the potential of the other charges. So going to calculate the equilibrium points, these will be very close to the basis charge. As a result, the schema of the next experiment would be too "small" if compared to the initial schema.

996 I. Bendato et al. One way to quantify the rate of convergence of the algorithm is to measure the size of schemes at each iteration. For N = 1 the measurement coincides with the length of the segment. If the convergence is too fast, the experimental domain would not be well explored by the algorithm leading to possible errors in the identification of the stationary point. Fig. 2 shows an example of initial schema in which it has a too rapid convergence of the algorithm: already at the first iteration the positioning scheme of the charges "collapses" in a neighborhood of the basis charge, which has a potential associated with it strongly superior to the others.

Basis Charge Identification

Identification of the farther equilibrium point

New Scheme

FIG. 2 Example of too rapid convergence in a mono-dimensional case

This problem was solved by the Authors by inserting in the formula for calculating the equilibrium point a parameter 1≥ω that allows to avoid a too rapid convergence of the algorithm. From a physical point of view ω represents an opposite force to the attraction ones between charges, generated by an external vector field (Fig.3). By setting 1=ω there are no external vector fields. On the contrary, if 1>ω it is set the intensity of the external force that allows to reach slowly the convergence; obviously values <1 are meaningless, because they would act as "accelerators" of convergence, not allowing a through exploration of the experimental domain on which the objective function is defined. The attraction formula is then modified as follows:

( )( )BASIS

i

CHARGEBASISiEQ

xfxf

rx i

ω+= −

1,

A correct calibration of omega therefore allows to solve a potentially wide range class of optimization problems.

Attraction force optimization 997

FIG. 3 Effect of the insertion of an external vector field ( 1>ω ) An automatic calibration of ω could be implemented by comparing the reduction, in percentage terms, of the size of the schema between one iteration and the other. If this percentage exceeds a threshold of tolerance set initially by the experimenter, ω is re-calibrated within the same experiment and the comparison between the basis particle base and the remaining particles that form the current schema could be repeated. In this way, the control that in the native version of AFO the experimenter could do only a posteriori observing the inclination of the curve of convergence, can now be conducted automatically by the algorithm. Anyway the Author suggest to set as the initial choice, a value of omega = 1. 2.2 AFO generalized to N-dimensional problems In optimization problems in which are involved more independent variables (N) the schema of each single experiment consists of a hyperrectangle defined by the lower-bound )( Lx and the upper-bound )( Ux where ],...,,[ 21 Nxxxx = . In correspondence of each vertex of the hyperrectangle identified is placed a charge. The algorithm then proceeds with the placement of the jm remaining charges, with j=1,….,N suitably equispaced on each dimension of the hyperrectangle. Note that, for each dimension, the experimenter can choose a different number of additional charges Nmmm ,...., 21 . Indicating with ix the single charge, the heuristic proceeds to the calculation of the potential )( ixf associated to each of them. Having determined the basis charge, it is compared with each of the other charges in the schema by the determination of the equilibrium points with the formula:

998 I. Bendato et al.

( )( )BASE

iiEQ

xfxf

rx

ω+

=1

2,

where:

2r distance (2-norm) is the distance between the basis charge in the schema and

the i-th charge ω calibration parameter of the heuristic The algorithm then proceeds in a similar way to the one-dimensional case. A procedural schema of the algorithm is reported in Figure 4. Figure 5 explains the behavior of AFO in the two-dimensional case for a particular starting configuration, with m = 1 and therefore a total number of particles in the initial scheme equal to 8. As can be noted at the first iteration is identified as the basis particle the particle A having a function evalution equal to 100. Then the algorithm proceeds with the calculation of the equilibrium points between the remaining particles and the basis particle identified in Fig.5 with the particles AB , AC, etc ... Among these equilibrium points is identified the point having the maximum distance with respect to the basis particle, in this specific example the particle AE. The algorithm then proceeds with the translation of a new schema delimited by the particle base A the equilibrium point AE, again characterized by a total number of particles equal to 8 maintaining the condition of m = 1. The Authors show, for a full discussion, some applications of AFO to functions

ℜ→ℜ2:f . Let consider the function ( ) )244(, 22 yxyxyxf +−−= . It has a global optimum in

]0,0[=OPTx and 0)( =OPTxf . AFO has been initialized with the following parameters: • Research Domain: 2]10,10[− • 1=ω • 181 =m • 182 =m • 30sexperiment of nummax = The results of the AFO application (Fig.6) are:

005--3.4038e)( _ =AFOOPTxf ]0058.0,0026.0[_ −−=AFOOPTx

Attraction force optimization 999

FIG. 4 Generalization of AFO procedure

FIG. 5 AFO behavior in 2-dimensional case

1000 I. Bendato et al.

FIG. 6 AFO iterations: experiments and schema The Matlab code developed by the Authors for implementing AFO presents the results in the form referred to Figure 7 in which: • the first graph in the top left represents the trend of the average convergence of x and y as a function of the number of experiments (budget); • the second graph in the upper right shows the trend of the potential associated to the basis charge as a function of the number of experiments. Notice how even after the tenth iteration, the image of the basis charges in each experiment is seated around the value 0; • the graph at the lower left shows the convergence of the schema as a function of the number of experiments (axis of ordinates). Also in this case it is found that the measurement of the schema is "closed" around the optimal point already after the tenth iteration; • the graph in the lower right represents the surface of the function investigated.

FIG. 7 Output of Matlab code after AFO implementation: analysis of results

AFO does not gain any advantages from the fact that the optimum point is positioned at the center of the domain. By way of example let consider Schwefel function:

Attraction force optimization 1001 ( ) { } 31052,72max 2121 −−+−+= xxxxxf

2]100,100[−∈x ]100,100[=OPTx

310)( −=OPTxf Figure 8 reports the behaviour of AFO and demonstrates the capability of AFO to identify the optimum even on the corner of the search space.

FIG. 8 Output of Matlab code after AFO implementation: analysis of results 3. Tests on benchmark functions This section presents in detail the tests conducted on a set of widely used benchmark function to examine the performance of AFO in solving optimization problems. For each of the test functions considered, I different experimental domains were investigated, all centered in the global optimum, known from the literature, and whose amplitude was defined by the variableδ : ],[ δδ

OPTIMUMxI ∈ where ]10,5,1[=δ . In the following sections, these domains are respectively denoted by →→

± 1OPTIMUMx , →→

± 5OPTIMUMx , →→

±10OPTIMUMx . 3.1 Tests on 2-dimensional benchmark functions In order to conduct a correct validation of the AFO technique in the 2-dimensional case, AFO has been tested and compared with some traditional methods described in the first section of the paper using the following benchmark functions: McKinnon; Rastrigin; Schaffer; Salomon; Himmelblau; Schwefel; Griewank; De Jong's Fifth Function. Concerning AFO, it has been tested setting:

• 1=ω • 81 =m • 82 =m • Maximum number of function evaluations = 250

1002 I. Bendato et al. Regarding traditional techniques, it was decided to use those listed below, maintaining the parameters as set by default in Global Optimization Toolbox and Optimization Toolbox of Matlab:

• BFGS o TolFun = 610− o TolX = 610− o step-sizes for the gradient between 810− and 110−

• NELDER-MEAD SIMPLEX o TolFun = 410− o TolX = 410− o Reflection Coefficient: 1=ρ o Expansion Coefficient 2=χ o Contraction Coefficient 5,0=ψ o Shrinkage Coefficient 5,0=σ

• GPS 2N o TolFun = 610− o TolX = 610− o MeshContraction = 5,0 o MeshExpansion = 2

• Genetic Algorithms (GA) o Tol = 610− o Elite Children= 2 o Cross – over = 8,0 o Number of generations = 25 o Number of individuals at each generation = 10

• Simulated Annealing (SA) o cooling scheme of exponential type and initial temperature

1000 =T . By way of example, Figures 9 and 10 show the behavior of AFO with reference respectively to the function of Schaffer and De Jong's Fifth Function.

. FIG. 9 AFO applied to Schaffer function

Attraction force optimization 1003

FIG. 10 AFO shrinkage around the global optimum of the De Jong's Fifth Function ( 1=ω )

Tables 1-4 present the results obtained in the testing phase on 2-dimensional functions. In particular, inside each cell of the tables are reported: the optimum point obtained by applying the algorithm, and the relative error committed. The error was calculated as the distance in terms of 2-norm between the exact coordinates of the optimum point of the surface considered, 1,OPTIMUMx and

2,OPTIMUMx , and the coordinates 1x and 2x obtained by the different heuristics:

( ) ( )222,2

11, , xxxxerror OPTIMUMOPTIMUM −+−= In each table, the last column on the right shows the average error committed by each heuristic in the three different experimental domains considered. It should be noted that, concerning Genetic Algorithms and Simulated Annealing, given the stochastic nature of these techniques, five experimental campaigns have been conducted. Consequently, in terms of point of maximum response, the mean values and the respective standard deviations are reported. It can be seen that AFO has been the methodology capable of committing the smallest average error. In addition, the tests carried out show that GA, under equal conditions, represent the heuristic that most closely matches the results obtained with AFO. However, it can be noted that, in the case of GA, the error is estimated as the average of 5 simulations and is therefore characterized by a standard deviation. Since the values of the standard deviation are non-negligible it seem possible to conclude that the GA have not yet reached a satisfactory convergence to the result.

1004 I. Bendato et al.

Table 1 AFO vs traditional techniques: MCKINNON function (N=2)

Table 2 AFO vs traditional techniques: RASTRIGIN function (N=2)

Attraction force optimization 1005

Table 3 AFO vs traditional techniques: SCHAFFER function (N=2)

Table 4 AFO vs traditional techniques: GRIEWANK function (N=2)

3.2 Tests on N-dimensional benchmark functions The second validation step was the application of AFO to N-dimensional benchmark functions.

1006 I. Bendato et al. In particular, the Authors took into consideration functions having a number of independent variables equal to N = 10. The results obtained with AFO were compared, like in the case with N = 2, with those resulting from the application of some traditional techniques. In particular the test functions considered for this purpose were: • Salomon • Schwefel • Rastrigin

in the domain →→

± 5OPTIMUMx . Regarding AFO it was decided to set, for each test function, a maximum number of four iterations. Each iteration of the algorithm presented 1184 charges. Other techniques have been tested using the parameters already presented for the case N = 2 and keeping the number of function evaluations coherent with AFO. The results obtained are reported in Tables 9-11. In correspondence with each algorithmic technique Table 9-11 show the coordinates of the optimum point determined and, in the last column to the right, the error committed based again on the 2-norm distance. The AFO was further validated, using also the other benchmark functions like in the 2-dimensional case, obtaining good performance. However in order not to burden the discussion, the Authors have decided not to add them in the article. Similarly to the case N = 2, for the techniques of stochastic nature (GA and SA) five experimental campaigns were performed. In each cell of the tables has been reported iiaveragex σ±,

, recalling that iσ is the standard deviation of the i-th component with respect to its average value. The author point out that the methodology could have computational problems in the optimization of objective functions with a large number of independent variables (N>10). AFO in fact requires the construction of geometric patterns in N dimensions: considering only the vertices, these grow exponentially with the size of the vector space of reference (2N). Let consider, by way of example, the minimization of the ninth benchmark function of the collection CEC2005 (Shifted Rastrigin’s Function) [29], whose 2D profile is reported in Figure 11.

FIG. 11 2D Profile of Shifted Rastrigin’s Function

Attraction force optimization 1007 Matlab allows to monitor the time taken for the construction of 100 patterns as a function of the vector space dimension (from N = 3 to N = 15). The calculation was made using a PC with eight processors, 32-bit, 3.45 GB of RAM. In Table 5 are reported the values [in seconds] from the test.

Table5 Computational time comparison

N 3 4 5 6 7 8 9 10 11 12 13 14 15 [sec]

0.825

0.9366

0.9539

1.159

1.6979

3.1109

4.999

9.119

17.18

33.1

67.92

140.89

299.91

We underline however that, regardless of the process time, in all cases, the algorithm AFO has successfully isolated the optimum point (see tables 6, 7 and 8).

Table 6 Error AFO vs traditional techniques: SCHWEFEL function (N=10)

AFO (ω =1) BFGS NELDER -

MEAD simplex GPS 2N SA GA

0 0,0044 0,0039 0,0032 2,2828 1,7288

Table 7 Error AFO vs traditional techniques: SALOMON function (N=10) AFO (ω

=1) BFGS NELDER - MEAD simplex GPS 2N SA GA

1,9514 15,811 15,9975 14,9974 8,022 2,4267

Table 8 AFO vs traditional techniques: RASTRIGIN function (N=10)

AFO (ω =1) BFGS NELDER -

MEAD simplex GPS 2N SA GA

2,2514 0 15,7313 0 4,8386 2,3038 4. Conclusion The Attraction Force Optimization (AFO) has proven to be an heuristic absolutely competitive with the search algorithms currently in use. The validation of the technique by using the main benchmark functions has highlighted both the effectiveness and efficiency of the algorithm, even taking into account the prefixed and limited budget of function evaluations. The strength of the methodology is certainly the ability to explore the entire domain combined with a sufficiently high speed of convergence to the stationary

1008 I. Bendato et al. point. The limitation is linked to the number of independent variables that the algorithm is able to manage. In particular, the algorithm has proved competitive with other traditional search algorithms on vector spaces up to 10 dimensions. Such characterization makes AFO particularly suitable also to the study of objective functions such as those that often characterize optimization problems in simulation where the variable involved are limited but the computational time for the objective function calculation is very high. For this reason, the Authors are starting to test the algorithm in the context of simulation models, in particular application of Discrete Event Simulation (DES) and Monte Carlo Simulation [4-6]. In these problems only the objective function's domain is known a priori while the function formula is unknown. References [1] C. Audet, J.E. Dennis Jr., Analysis of Generalized Pattern Searches, SIAM

Journal on Optimization, 13 (2003), no. 3, 889-903. http://dx.doi.org/10.1137/s1052623400378742 [2] M. Avriel, Nonlinear Programming: Analysis and Methods, Dover

Publishing, 2003. [3] R.R. Barton, J.S. Ivey, Modifications of the Nelder-Mead Simplex Method

for stochastic simulation response optimization, Winter Simulation Conference Proceedings, (1991), 945-953.

http://dx.doi.org/10.1109/wsc.1991.185709 [4] I. Bendato, L. Cassettari, M. Mosca, R. Mosca, F. Rolando, New Markets

Forecast and Dynamic Production Redesign Through Stochastic Simulation, International Journal of Simulation Modelling, 14 (2015), no. 3, 485-498. http://dx.doi.org/10.2507/ijsimm14(3)10.307

[5] I. Bendato, L. Cassettari, R. Mosca, P.G. Giribone, Monte Carlo method for

pricing complex financial derivatives: An innovative approach to the control of convergence, Applied Mathematical Sciences, 9 (2015), no. 124, 6167-6188. http://dx.doi.org/10.12988/ams.2015.58552

[6] I. Bendato, L. Cassettari, R. Mosca, F. Rolando, Improving the Efficiency

of a Hospital ED According to Lean Management Principles through System Dynamics and Discrete Event Simulation Combined with Quantitative Methods, Chapter in Intelligent Software Methodologies, Tools and Techniques, (2015), 555-572.

http://dx.doi.org/10.1007/978-3-319-22689-7_43

Attraction force optimization 1009 [7] R. Chiong, Nature-Inspired Algorithms for Optimisation, Studies in

Computational Intelligence, 193, 2009. http://dx.doi.org/10.1007/978-3-642-00267-0 [8] A.R. Conn, N.I.M. Gould, Ph.L. Toint., A Globally Convergent Augmented

Lagrangian Algorithm for Optimization with General Constraints and Simple Bounds, SIAM Journal on Numerical Analysis, 28 (1991), no. 2, 545-572. http://dx.doi.org/10.1137/0728030

[9] A.R. Conn, N.I.M. Gould, Ph.L. Toint, Trust-Region Methods, MPS/SIAM

Series on Optimization, SIAM and MPS, 2000. http://dx.doi.org/10.1137/1.9780898719857

[10] J.E. Dennis Jr, R.B. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Prentice-Hall Serie in Computational Mathematics, Prentice-Hall, 1983. http://dx.doi.org/10.1137/1.9781611971200

[11] R. Fletcher, M.J.D. Powell, A Rapidly Convergent Descent Method for Minimization, The Computer Journal, 6 (1963), 163-168. http://dx.doi.org/10.1093/comjnl/6.2.163

[12] R. Fletcher, Practical Methods of Optimization, John Wiley and Sons, 1987.

[13] Z.W. Geem, J.H. Kim, and G.V. Loganathan, A new heuristic optimization algorithm: harmony search, Simulation, 76 (2001), 60-68.

[14] W.E. Gettys, F.J. Keller, M.J. Skove, Fisica Classica e Moderna, McGraw

Hill, 1998. [15] P.E. Gill, W. Murray, M.H. Wright, Practical Optimization, London,

Academic Press, 1981.

[16] J. H. Holland, Adaptation in Natural and Artificial Systems, MIT Press, 1975.

[17] S. Kirkpatrick, C.D. Gelatt Jr., M.P. Vecchi, Optimization by Simulated

Annealing, Science, 220 (1983), 671-680. http://dx.doi.org/10.1126/science.220.4598.671

[18] M. Mahdavi, M. Fesanghary , E. Damangir, An improved harmony search algorithm for solving optimization problems, Applied Mathematics and Computation, 188 (2007), 1567–1579. http://dx.doi.org/10.1016/j.amc.2006.11.033

1010 I. Bendato et al.

[19] R. Mosca, P. Giribone, G. Guglielmone, The simplex method in the optimization of discrete simulations of complex industrial plants: theory applications comparisons, International Journal of Modelling & Simulation, 6 (1986), no. 2, 58-64.

[20] R. Mosca, P. Giribone, The Nelder-Mead method: a better sequential technique to get optimum region in discrete simulation of complex plants, Proceeding III IASTED International Conference Modelling and Simulation, Lugano, Switzerland, 1985.

[21] J.A. Nelder, R. Mead, A Simplex Method for Function Minimization, The Computer Journal, 7 (1965), 308-313. http://dx.doi.org/10.1093/comjnl/7.4.308

[22] A. Rykov, Simplex methods of direct search, Engineering Cybernetics 18, 1980.

[23] A. Rykov, Simplex Algorithms for Unconstrained Optimization, Problems of Control and Information Theory, 1983.

[24] D.F. Shanno, Conditioning of Quasi-Newton Methods for Function Minimization, Mathematics of Computing, 24 (1970), 647-656. http://dx.doi.org/10.1090/s0025-5718-1970-0274029-x

[25] K. Socha, M. Dorigo, Ant colony optimization for continuous domains, European Journal of Operational Research, 185 (2008), 1155-1173. http://dx.doi.org/10.1016/j.ejor.2006.06.046

[26] J.A. Snyman, Practical Mathematical Optimization: An Introduction to Basic Optimization Theory and Classical and New Gradient-Based Algorithms, Springer Publishing US, 2005. http://dx.doi.org/10.1007/b105200

[27] T. Steihaug, The Conjugate Gradient Method and Trust Regions in Large Scale Optimization, SIAM Journal on Numerical Analysis, 20 (1983), 626-637. http://dx.doi.org/10.1137/0720042

[28] T. Strutz, Data Fitting and Uncertainty (A practical introduction to weighted least squares and beyond), Vieweg and Teubner, Wiesbaden, Germany, 2011.

[29] P.N. Suganthan, N. Hansen, J.J. Liang, K. Deb, Y.P. Chen, A. Auger,

S. Tiwari, Problem Definitions and Evaluation Criteria for the CEC 2005 Special Session on Real-Parameter Optimization, Technical Report, Nanyang Technological University, Singapore and KanGAL Report Number

Attraction force optimization 1011 2005005 (Kanpur Genetic Algorithms Laboratory, IIT Kanpur), 2005.

[30] M. Thakur, S.S. Meghwani, H. Jalota, A modified real coded genetic algorithm for constrained optimization, Applied Mathematics and Computation, 235 (2014), 292–317. http://dx.doi.org/10.1016/j.amc.2014.02.093

[31] V. Torczon, On the Convergence of Pattern Search Algorithms, SIAM Journal on Optimization, 7 (1997), no. 1, 1–25. http://dx.doi.org/10.1137/s1052623493250780

[32] F.H. Walters, L.R. Parker, S.L. Morgan, S.N. Deming, Sequential Simplex Optimization, CRC Press, Boca Raton, Florida, 1991.

[33] Y. Wu, S. Zhang, A self-adaptive chaos particle swarm optimization algorithm, Telkomnika (Telecommunication Computing Electronics and Control), 13 (2015), no. 1, 331-340.

http://dx.doi.org/10.12928/telkomnika.v13i1.1267 [34] C.J. Xia, Y. Zhou, H.Y. Huang, A chaos particle swarm optimization

algorithm for optimal power flow, Advanced Materials Research, 1008-1009 (2014), 466-472. http://dx.doi.org/10.4028/www.scientific.net/amr.1008-1009.466

[35] X.-S Yang, Nature-Inspired Metaheuristic Algorithms, Luniver Press, UK,

2008.

[36] X.-S Yang, A New Metaheuristic Bat-Inspired Algorithm, Nature Inspired Cooperative Strategies for Optimization (NISCO 2010), (Eds. J. R. Gonzalez et al.), Studies in Computational Intelligence, Springer Berlin, 284, 65–74, 2010. http://dx.doi.org/10.1007/978-3-642-12538-6_6

[37] Z. Yang, K. Tang and X. Yao, Self-adaptive Differential Evolution with Neighborhood Search, Proceedings of the 2008 IEEE Congress on Evolutionary Computation (CEC2008), Hongkong, China, 1110-1116, 2008. http://dx.doi.org/10.1109/cec.2008.4630935

[38] J.Q. Zhang, A.C. Sanderson, JADE: adaptive differential evolution with

optional external archive, IEEE Trans. Evol. Comput., 13 (2009), no. 5, 945-958. http://dx.doi.org/10.1109/tevc.2009.2014613

Received: December 5, 2015; Published: March 20, 2016