Characterizing and Evaluating Problem Solving Self-* Systems

Elsy Kaddoum, Marie-Pierre Gleizes, Jean-Pierre Georgé IRIT - Université Paul Sabatier

Toulouse - France {kaddoum, gleizes, george}@irit.fr

Gauthier Picard SMA@G2I – Ecole Nationale Supérieure

des Mines de Saint-Etienne - France picard@emse.fr

Abstract—In the last few years, the growing complexity of current applications has led to the design of self-organizing systems presenting self-* properties. Those systems are composed of several autonomous interactive entities. They behave autonomously and present interesting characteristics allowing them to handle dynamics coming from exogenous and endogenous changes. As those systems are different from classical ones, new or updated characterization and evaluation criteria are required for analyzing the contribution of self-* properties and system performance.

This work is a first step for evaluating self-* systems. Its main purpose is to propose a simulation based evaluation framework which aims at guiding the evaluation of this kind of systems from the design phase all the way to the execution results: evaluation of the system at run-time, intrinsic and development methodologies characterization.

Keywords - self-* systems characterization; self-*systems evaluation; self-*systems comparison.

I. INTRODUCTION

In the last few years, the growing complexity of current applications has led to design self-organizing systems presenting self-* properties. Those systems are necessarily composed of several autonomous interactive entities (called agents in this paper) acting into an environment. In general, the global behavior of the system emerges from the local interactions between its agents. Those systems behave autonomously and must handle the dynamics coming from exogenous or endogenous changes so, they are able to self-adapt [1][2]. To clarify, endogenous changes are perturbations caused by entities belonging to the system. Exogenous changes solely occur from entities outside the system. Handling these perturbations can lead to change agents behavior or even the global objective of the system.

Moreover, those systems are characterized by different self-* properties such as:

• Self-organization: the mechanism or the process enabling a system to change its organization without explicit external command during its execution time [3].

• Self-stabilization or homeostasis: the system always reaches a stable state [4].

• Self-healing: the mechanism that allows a system to detect and recover from errors.

• Self-tuning: the system is able to adjust its parameters.

Generally, those characteristics make systems more robust than classical ones and by that more adequate to complex and real-life problems common in all the domains: telecommunications, economics, bio-informatics, industrial environments etc. Real-life problems present a lot of dynamics where events are unpredictable and can occur at different steps. Classical systems with no self-* properties, are usually weak to solve such problems.

The main advantages of classical systems over self-* systems are that, for the most of them, formal proofs of reaching good solutions exist and the functioning of the system can be validated without deploying it in a real environment. Classical systems are usually functionally adequate (e.g. they do what they are designed for). The question is how can we point out the advantages of self-* systems over classical ones even that the most of them are not deterministic? Or, how can self-* properties guarantee good solutions under perturbations? Or, how can we proof that they are functionally adequate? An important limitation when trying to ascertain that a self-* system behaves well, comes from its main difference with classic system: its ability to change, adapt, and evolve. Both the computational complexity and decentralization might completely change when a system is subject to important modifications in its structure, the way it works, its communication, and so on. Currently, there is no tool to formally prove the behavior of self-* systems [5][6][7] and only empirical evaluation can help answering those questions.

For the evaluation of those systems, researchers commonly use a much reduced number of criteria, such as execution time or communication load. These two criteria are important but not sufficient to highlight their specificity. Other criteria such as homeostasis, decentralization degree (a measure of the control and decision making distribution between agents [33]) or complexity are more appropriate

2009 Computation World: Future Computing, Service Computation, Cognitive, Adaptive, Content, Patterns

978-0-7695-3862-4/09 $26.00 © 2009 IEEE

DOI 10.1109/ComputationWorld.2009.100

137

and must be studied. In this paper, we aim at collecting a number of thoughts and ideas about the task of how evaluate such systems.

As those systems are different from classical ones, new or updated characterization and evaluation criteria are required for analyzing the contribution of self-* properties and system performance. Generally, these systems show two kinds of behavior: the nominal behavior (N) which is the normal behavior (i.e., the behavior of classical systems) and the self-* behavior (S) which enables the system to handle unexpected dynamics. In some systems, those behaviours are distinct. In some others, they are intrinsic. This paper aims at guiding the evaluation of this kind of systems from the design phase to the execution phase by providing different sets of main criteria.

In the following, we classify the identified evaluation criteria in three parts: the first concerns the system at runtime, the second discusses its intrinsic characteristics, and the third presents the methodologies characterization. In each part, we present the main criteria for measuring, by defining it, and trying to give a way to examine it. Finally, we propose to compare a system with other systems to point out its originality, advantages and weaknesses.

II. EVALUATION OF THE SYSTEM AT RUN-TIME

Different questions are to be considered by designers to validate the well-functioning of the system and of the self-* mechanism, such as: is the system able to solve the problem for which it is conceived? Is the system able to self-adapt in an efficient way? Are the self-* properties implemented in the system sufficient to insure the robustness? Etc. Evaluating the system at run-time concerns the results obtained at the end of its execution and its behavior during it. The former part gives information on the system performance, while the latter part enables the study of how the system handles dynamics and points up the self-* characteristics. Note that the end of the execution of a self-adaptive system is not so easy to determine. This kind of systems has to continuously self-adapt to its environment and has not to stop. If it has to reach a given state or a solution, because of the distribution, the end cannot be known by the system itself. Designers must add a global observer of the system which can decide if the system has reached its goal and by that has to stop.

A. Performance metrics

The evaluation criteria presented in this section concerns the results of different system executions for a given scenario. We consider six points which are as follows: time, communication load, non determinism, accuracy/quality, progress and memory usage. Once these measures are done, they can be used by the evaluator to compare its system with others.

Time. Commonly utilized, time has different definition. In [8], it is expressed by the number of checked constraints, or more generally the number of performed atomic operation. Another frequent way is the measurement of computing time (CPU) [9][11]. For this, the characteristics of machines should be considered. [12] introduces some transformation ratios in order to make run-times of different machines comparable. Others authors study the time as the number of steps needed by agents to reach the solution, for example in [9], it is calculated as the fabrication cycle number. While in [13], it is the number of rounds for synchronous algorithm. Another common way is the time needed by the most costly operations.

We consider that the number of steps needed by agents is the most appropriate and easiest way to measure time. In fact, considering the measurement of CPU needs to take into account machines and platform characteristics, while counting the number of verified constraints in distributed context must consider the asynchrony between the agents [8], which is also the case of considering the time needed by the most costly operations.

At the beginning of distributed systems, designers worried that the systems spend too much time communicating than working. Currently in self-* systems, the dilemma has shifted to the ratio between working and self-adapting. In self-* systems, we propose to distinguish when it is possible the nominal time NT needed by the self-* system to perform its usual function (which could be to solve a problem) from the self-adapting time ST needed by the self-* system to self-adapt to changes in the environment. For example, in a mail server representing self-healing characteristics we can distinguish the time spent by the server doing its nominal behavior (treat sent or received mail messages) from the time spent on searching for help when it is overloaded. When this distinction is not possible, we propose to evaluate the Mean of the Time Differences (MTD) between the execution of the system in a scenario without perturbation and its execution in a scenario with perturbations. For instance, in a self-* system solving a constraint satisfaction problem it is difficult to perform this separation. Some can consider that the time spent by an agent to find an affectation for its variable is the NT, while others consider it as the ST.

In some problems such as airport planning, users of self-* systems need to reach the solution in a very short time. In such cases, the time criterion is more important than the others.

Communication Load. In self-* systems, agents usually need to communicate in order to reach the solution or the organization needed to attain the solution. Examining the communication load enables the study of network load

138

which is a very important criterion when deploying the system in a real environment.

A way to measure this parameter is the amount of exchanged messages. The exchanged messages can be sorted in two sets: the nominal messages NCL for the solving part and the self-adapting messages SCL for self-adaptation. For example, in a mail server representing self-healing characteristics we can distinguish between sent or received mail messages and messages asking for help when the server is overloaded.

But, as the number of exchanged message is highly dependent on centralization/decentralization degree (Calculated as the ratio between the number of agents in charge of the decisions and the total number of agents) of the algorithms, it would be interesting to analyze the communication load in regard to the decentralization degree. In decentralized algorithms more messages are needed among agents while in centralized approach agents don’t have to exchange a lot of information. Unfortunately, rare are the studies that combine them.

Different other parameters are interesting to examine when studying the communication load [14]. We distinguish between them the message size and complexity which are relevant indicators of the time needed to understand and process messages. The network load is mainly calculated using this communication load in addition of transit and latency time.

The study of this criterion progression is necessary when the number of communicating agents is important. In the example of the mail server considered above, we have a lot of agents representing different points of the network. In addition to this, the number of send and received messages between users is important. Having a lot of SCL will lead to overload the network and by that to complicate the problem than solving it. That is why in these cases, the self-* system must use the less possible messages for self-adaptating.

Accuracy/Quality. The accuracy or quality of solution refers to functional adequacy of the designed system. In [7], a system is functionally adequate if it is useful for its environment [15]. A convenient measure for solution quality is the distance from the optimal value δ(Scur; Sopt). The main problem of this measure is the determination of the optimal solution. In some cases, this value is known for some instances, but in general it is unknown. That’s why some authors usually use scenario belonging to well-known benchmark and compare their solution with best known solutions. The quality of self-adaptation can be highlighted when running the system on a sufficient number of scenario presenting different levels of dynamics and unpredicted events. The aim is to show that the system maintains a high quality level. This criterion is important for any self-* system under any type of perturbation.

Non Determinism. In self-* systems, information are distributed among agents which behave asynchronously. These characteristics normally make the resolution not deterministic. As a consequence, for one considered scenario, the solutions obtained from one execution to another may be different [16]. Studying the distances between those obtained solutions give information about the stability of the system in comparison with exact approach. Usually authors use measures of central tendency such as mean or median and variability such as standard deviation and variance to examine the non determinism of a system [17]. As the Accuracy/Quality criterion, this criterion must always be studied as it indicates the stability of the self-* system and allow designers to compare their approaches with exact ones.

Progress. Evaluating the progress NP of the system refers to the study of how the system progressively reaches a complete solution. A measure of this could be the percentage of the main goal reached at each step. To do this measure it is not necessary to know the optimal solution. It is just the measure of the activity. For example, in [9], the progress is measured as the total number of finished containers at each step. During time, this can be observed by looking at the curves representing this progress as they will have specific forms (linear, exponential, and logarithmic) (see Section V). More interestingly for self-* systems, we can analyze how self-adaptation influences this progress: does it maintain or even increase the rate (which is an ideal situation), does the progress slow down (how much?), or even worse, is there a decrease in the progress (the worst being a reset to zero of the progress). Finally, we can also evaluate SP, the progress of the self-adaptation following a perturbation in the same manner as NP.

Memory Usage. Another interesting uncommon criterion is the amount of memory used by each agent for the solving part NM and the self-adaptation part SM. This criterion has a direct influence on the computational complexity (see Section III-A), when, for example, at each step, an agent must process information from all other agents or just from a local neighbourhood. The deployment of the system in real-world application is also affected by a high usage of the memory. For example, in the context of ambient intelligence, where small devices are usually constrained by small memory capacities, the memory usage is more important than the time criterion [32].

B. Homeostasis and Robustness

As stated above, the main property of self-* systems is their ability to maintain their functioning in an environment presenting a high level of dynamics. In other

139

words, we can say that self-* systems possess robustness, and homeostasis/self-stabilization abilities. The robustness ability is presented by their capacity to maintain their behavior when perturbations occur [1]. The homeostasis ability is what enables the system to regain its normal behavior after being perturbed. In [4], authors defined the homeostasis as the capacity of regaining an ideal state in which the system is operating in a maximally efficient way. Studying those two abilities enables to understand in depth how the system handles the dynamic changes of the environment, and by that validate its adequacy to solve the problem. That is why they must be studied for every self-* system.

Robustness. Considering dynamics is a very interesting point in complex problem solving. Events occur at different steps and are usually unpredictable. A robust system is a system which is able to maintain a stable behavior even under perturbations. To measure the robustness, we can consider:

• The system maintains a functional adequacy even during perturbation. In other terms, the variation of the solution quality is weak under perturbations (ΔQ < ε).

• The new solution (state) reached by the system is closed to the previous one.

• The amount of changes inside the system between the state when the perturbation occurs and the new stable state is minimal.

Time for Adaptation. The time for adaptation is the time needed by the system to regain its normal behavior after the occurrence of a perturbation. When measuring the time (see Section II-A), we introduce ST the time for self-adaptation which represent the time needed by the system to self-adapt along its execution. While here, we study the time STAlc needed to take into account one type of dynamic changes. The number of steps needed by agents to return to the normal functioning is an appropriate measure of this criterion. Two levels are to be considered to measure adaptation time: local level and global level.

In the first one, the number of steps needed by each agent is examined. Measuring this time allows the designer to examine the self-adaptability of the agents. In the second one, the self-adaptivity of the global behavior of the system is studied by measuring the number of steps needed by the system to regain its normal behavior (see Figure 3).

Measures for run-time evaluation

• Time for self-adaptation ST, time for solving NT, time for adaptation STAlc or the mean of the time differences MTD.

• The number of messages for solving NCL and the

number of messages for self-adaptation SCL. • The quality of the solution (δ(Scur; Sopt)) and the

degree of non determinism of the resolution. • The progress of the resolution NP and the progress of

the self-adaptation SP. • The memory usage rate for the nominal behavior NM

and the self-* behavior SM. • The robustness level of the system (ΔQ < ε).

III. INTRINSIC CHARACTERIZATION OF THE SYSTEM

The main limit of the evaluation criteria presented in the previous section is that they can only be measured during or after the execution of the system. Since this can only be done on a limited number of scenarii or situations, these evaluations cannot guaranty the qualities of the system in every real-world situation and on the long run.

This section completes the previous one by defining, discussing and presenting as evaluation criteria several characteristics of self-* systems which can be used without having to actually run the systems. Given the knowledge of the nature of a system, of its functioning, we can partly study its computational complexity, the decentralization, the requirements of the local algorithm or the influence of the number of agents and their computation needs. This enables to already evaluate and compare such systems.

A. Computational Complexity

Very early in Computer Science [18], the need arose to be able to measure the time or memory required by a program or algorithm. Basically, by looking at a classic algorithm, one can ascertain how the execution would fare in a worst-case scenario and by expanding the set of input data to a real-world problem; one can predict that the algorithm would fail to give a response in a reasonable time. This problem of scalability has been thoroughly studied in the last 40 years and a formal theoretical framework has been given [19]. With it come the well-known but difficult to completely grasp notions of orders of complexity and NP-hardness. The main idea to keep in mind is that we currently have problems which we do not know how to solve in a polynomial time. The general theory is quite out of the scope of this article but the question remains when evaluating a self-* system: can we know how efficient it is by looking at the code? If the system attains its self-* properties via a global control algorithm, it perfectly fits the previous framework and can thus be studied and evaluated by its tools. For evaluation purposes, this is an interesting advantage, but the major drawback is that global algorithms which solve problems with exponential complexity present a lot of limits such as

140

time and memory consuming. Self-* systems usually rely on different local algorithms cooperating together and influencing each other. Thus, the computational complexity is difficult to measure at the global level and must be studied at the local one by considering the influence of each local action on the rest of the system.

B. Decentralization and Local Algorithms

In our experience, nearly all systems exhibiting self-* properties on real-world complexity scale problems rely in one way or another on distribution of the computation, decentralization of the control and local mechanisms to explore the search space. When one local, even simply calculated (in terms of complexity) change can have non-linear repercussion throughout the whole system, how can we calculate the computational complexity of the whole system? There exists work [20] on complexity theories for local search algorithms, with for instance the definition of the PLS (Polynomial Time Local Search) class, but they fix very strict boundaries. The computational complexity can be studied by measuring the local complexity of each sub-algorithm, the decentralization degree, the influence of each local action and the scalability of the system. Analyzing these criteria is important to know in which situation the system will fail to reach a solution or to predict the performance of the system and by that improve it.

Local Computational Complexity. Pending on a breakthrough in Theory of Complexity, we are thus reduced to analyze the local algorithm by itself. This is already a criterion to disqualify certain local algorithms which require too much computational power.

Decentralization. But we can complete this with an evaluation of some sort of ratio between the decentralization and the amount of local computation needed. If the system has limited decentralization, it may fall in the NP-complete problem class. If the decentralization is important, but the local computation is complex, then each local decision may be NP-complete. In short, how many, how decentralized, how simple are the agents in the system and how many other agents does it need to know or interact with?

Bear in mind that this criterion can not be used alone as the least complex local algorithm would be for instance to always choose a random action for an agent, but would certainly produce one of the worst systems ever. On the other hand, it can be usefully applied to filter local algorithms growing too much in complexity. For instance, an algorithm where an agent has to check all the properties of all the other agents of a system, and with a prediction for n steps ahead depending on its chosen action, is a very naive approach.

Action Locality. In the same way, we can also add our understanding of the repercussions of a local action on the rest of the system. Since self-* systems are complex systems, they are prone to the well known butterfly effect as we can observe in systems which use gossip mechanisms or propagation in neural networks [21]. But we can already evaluate how probable this will be by studying the strategies of the agents in regard to this. For instance, does an agent systematically broadcast its information and needs? Or does it proceeds by selecting its targets based on an evaluation of their relevance? Another example would be the min-conflict [22] types of heuristics which try to progress in the solving by minimizing the negative impact inside a neighbourhood, and by consequence, reduce the overall complexity.

Initial and Acquired Knowledge. Some algorithms require specific or extensive knowledge to be provided to the system before it can even start. For instance, in many meta-heuristic approaches like ACO or PSO, a preliminary analysis of the solution space is required to set up parameters like the number of self-* agents or the probability distribution characterizing their behaviours [23]. In the same way, some algorithms require this during their activity to be able to be efficient. This can be evaluated and used as a criterion as this kind of systems is more difficult to deploy.

Agent Number Influence. The number of agents in a system can drastically increase or decrease the efficiency of the system. This is an important factor to ascertain the optimal organization, for instance when deploying satellite swarms [24]. More importantly for self-* systems, it enables to judge the scalability of a system as it evolves during its life. This can be, of course, evaluated during run-time, but also by analyzing how the system is built: Does it facilitate the introduction of new agents? Can agents easily delegate? Are extensive self-organizing abilities present?

Local Complexity and Decentralization Criteria

• Calculate the computational complexity of the local decision process.

• Evaluate how decentralized the system is. • Evaluate how much influence the action of one

agent potentially has on the rest of the system. • Evaluate the difficulty to provided required initial

knowledge to the systems or the acquisition of it during execution.

• Analyze the scalability of the system.

141

IV. DEVELOPMENT METHODOLOGIES CHARACTERIZATION

Another perspective to analyze before developing a given distributed self-* system is the development methodology, i.e., the process to follow and the concepts to manipulate during the development phases (analysis, design, implementation, etc.). In fact, all the approaches one can consider to develop a self-* system, are not equal from this point of view. However, even if we can try to define criteria concerning this perspective, highlighted points are more qualitative than quantitative. Some analysis on how a self-* systems has been developed, are useful for next developments. Different studies have been realized for classical methodologies. Our aim is to underline the criteria dedicated to the self-* properties such as: identification of autonomous agents, distribution, generic points and flexibility.

Analysis and Identification of Autonomous Agents. During the analysis of a problem to solve, many factors may make the task tough. In some cases the development is straightforward while in some others it asks for some expert knowledge. One difficult task for the development of self-* system is the agents identification. In some methodologies, the problem specification leads to straightforward to this identification. While in some others, some analysis phases must be done. For example, in problem solving (using CSP framework), developing a DisCSP solver like distributed breakout (DBA) [25], ERA [26] or approaches like ant colony optimization (ACO) [27] or particle swarm optimisation (PSO) [28], only requires transforming the problem to solve into the constraint satisfaction problem framework. This can be a difficult task especially that choices made during this phase may strongly impact on the performance of the system and some parameters requiring expertise knowledge must be fixed. Another kind of approaches may not directly match a problem into a given formal or semi-formal framework. For instance, the AMAS (Adaptive Multi-Agent Systems) approach [29] focuses much on the properties or behavior an agent may have (i.e. cooperativeness) and therefore the designer will have to find what are the agents within the problem that can be agentified as to implement a self-* solver.

Distribution and Deployment Ease. All the developed systems are not easily distributable depending on the paradigms they rely on such as communication. We distinguish between two types of communication: stigmergic ones and direct ones. Stigmergic and swarming self-* approaches, like ACO, require the development or the use of specific environmental framework, such in tuple spaces middleware like the SwarmLinda [30] framework or the TOTA middleware [31] that can support the pheromone deposits, updates

and monitoring. Direct communication approaches require dedicated frameworks like Jade [34] or MAY [35] to be deployed. This also requires developing the entities as agents and programming them using dedicated libraries.

Additional effort must be done to deploy such systems. The evaluation of these efforts underlines the fact that the methodology is easily used by any designer or requires specific knowledge.

Generic points. Self-* approaches may also differ concerning their level of generic, i.e., the provided effort to adapt a method to a given problem. For instance, DisCSP approaches like DBA or ACO are completely generic, in the sense that they provide distributed solvers for problems as soon as they are expressed as CSP. However they still require some adjustment by setting parameters, like the time limit for DBA, the probability distribution of behaviours for ERA, or the entire set of pheromone and collective parameters for ACO. This criterion allows the disqualification of ad-hoc methodologies (conceived for a specific problem) compared with generic methodologies used for different types of problem.

Development Methodology Evaluation Criteria

• How much effort is required to adapt the general concepts of the method to a specific problem?

• How many effort is required to distribute a problem among a collective of self-* entities?

• How much effort is required to deploy the developed system?

• How large is the scope of problems for which the method is relevant or applicable?

• How difficult is it to add new constraints and problems, from an existing developed system?

V. CRITERIA USAGE FOR COMPARAISON

The characterization and evaluation of a system provide quantitative and qualitative values which enable to judge the system and to compare it with others in order to highlight its strengths and its limits. This section proposes means for comparing different systems using the criteria presented above and a panel of typical behaviours associated to their self-adaptation process. This last section is intended to illustrate how several of the previous criteria could be combined for evaluating self-* systems.

A. Global Comparative Evaluation

Evaluation of performance enables to show that a system is better or worse than another. The criteria to

142

compare two systems can be a subset chosen among the previous presented measures (see Section II, III, IV). Real systems are like we could see in Figure 1 where someone is the best on a dimension and not in another one. Those criteria are highly inter-independent and a multiobjective view is more appropriate. This multi-objective view cannot be reduced into a unique dimension combining all these criteria because they are very context sensitive, depending for instance on the application domain or initial requirements. Consequently, we propose a radar view as a relevant way to judge the functional adequacy of a self-* system. An “ideal system” is on the radar center because it outperforms the others in all the dimensions.

Figure 1. A radar chart to compare self-* systems

B. Some Performance Metrics Analysis

Still, each criterion, when analyzed in detail, can bring valuable information for comparing self-* systems. In the following, we give several illustrative curves representing a comparison between different systems. Each curve can be seen as representing a typical archetypal behavior for a given criterion or another.

Figure 2. How does the system reach a solution?

A first criterion that can be studied is the dynamic of adaptation of the system when a perturbation occurs during time, C= δ(Scur; Sopt) (see Section II-B). We distinguish between the non-adaptive systems and the self-adaptive systems. The former is not able to adapt its behavior under dynamics changes and by that can not improve its solution. For the latter, we differentiate between three types of response as seen on Figure 2:

• diagonal line when the system improves its solution in proportion to time,

• lower curve which is the best one as the system is able to rapidly adapt itself and reach a good solution.

• upper curve which represents a system needing a lot of time to self-adapt and attain a good solution.

In the same manner, this figure can be used to illustrate the response type of the system to one dynamic change (the event occurs at time t = 0).

Figure 3. How does the system adapt to change?

143

Figure 3 is used to illustrate the occurrence of different types of perturbations separately. If we consider the quality of solutions as Criteria C, we can see that at the beginning of the curves, the three represented systems improve their solutions. When the first perturbation P1 occurs, the distance to the optimal solution greatly increases for all of them. We can clearly see that a system with no adaptation then worsens its quality more and more with each new perturbation. An adaptive system recovers from the event.

Another information we can read on these curves, is the time for adaptation STAlc (see Section II-B). A self adaptive system is able to regain an optimal state more or less quickly (see full line and dashed line in Figure 3).

Another criterion that we can consider using the same figure, is the robustness of the system. For example here, the system represented by the full curve is more robust than the one represented by the dashed curve, as its reaction to perturbation is less important especially for perturbation P3. Moreover, this can also illustrate that some adaptive systems are able to learn from their adaptation to a type of perturbations so that whenever such a perturbation happens again, the system is ready to response efficiently.

A last point to consider is the number of messages SCL used for adaptation. We can see that a non-adaptive system maintains the same number of messages when perturbations occur (In figure 3, it is equal to zero and it is represented by the x-axis), while the self-adaptive systems increase SCL when perturbations occur and decrease it after adapting their behavior. The fact that the system represented by the full curve needs less messages than the one represented by the dashed curve could be explained by the decentralization degree in the system (see Section III-A).

VI. CONCLUSION

In this paper, we have presented a set of comparison criteria for self-* systems. We propose to observe and analyse them from three points of view:

1) Evaluate the system at runtime, when it behaves and is subject to endogenous and exogenous dynamics.

2) Evaluate the intrinsic properties of a system by focusing on the decentralization and the locality properties of the system, from its specifications.

3) Evaluate the methodology to develop the system from the modelling to the deployment phases.

Once collected, these criteria must be analyzed in global viewpoint. However, aggregating such disparate criteria (quantitative or qualitative) is not an easy task since they are highly inter-independent and a multi-

objective view is more appropriate. We discussed this point by illustrating some criteria with theoretical charts.

This work is a first step for evaluating self-* systems and highlighting the consequences of their specific behaviours devoted to their adaptation which is missing in classical systems. Nevertheless, we are aware that formal validation must be fundamental to enable the promotion of these systems in industries. Currently, because we cannot prove all the behavior of these systems, we think that partial proofs can be done and that a lot of works have to be done mixing simulation and formal classical proofs.

REFERENCES [1] P. Robertson, R. Laddaga, and H. Shrobe, “Introduction: the first

international workshop on self-adaptive software,” in Proceedings of the 1st IWSAS, ser. LNCS, R. L. . H. S. P. Robertson, Ed., vol. 1936. Springer, 2000, pp. 1–10.

[2] C. Bernon, M.-P. Gleizes, S. Peyruqueou, and G. Picard, “Adelfe: a methodology for adaptive multi-agent systems engineering,” in Third International Workshop on Engineering Societies in the Agents World (ESAW-2002), P. Petta, R. Tolksdorf, and F. Zambonelli, Eds., vol. 2577. Madrid, Spain: Springer-Verlag (LNAI), September 2002, pp. 156–169.

[3] M.-P. Di Marzo Serugendo, Giovanna and and A. Karageorgos, “Self-Organisation and Emergence in Multi-Agent Systems: An Overview,” Informatica, vol. 30, no. 1, pp. 45–54, janvier 2006, iSSN 0350-5596.

[4] R. P. W urtz, Organic Computing. Springer, 2008. [5] B. Edmonds and J. Bryson, “The insufficiency of formal design

methods - the necessity of an experimental approach - for the understanding and control of complex mas,” in AAMAS, 2004, pp. 938–945.

[6] B. Edmonds, “Engineering self-organising systems, methodologies and applications,” in Engineering Self-Organising Systems, ser. Lecture Notes in Computer Science, S. Brueckner, G. D. M. Serugendo, A. Karageorgos, and R. Nagpal, Eds., vol. 3464. Springer, 2005.

[7] M.-P. Gleizes, V. Camps, J.-P. Georg´e, and D. Capera, “Engineering Systems which Generate Emergent Functionalities,” in Engineering Environment-Mediated Multiagent Systems - Satellite Conference held at The European Conference on Complex Systems (EEMMAS), Dresden, Germany, 01/10/07- 05/10/07, ser. Lecture Notes in Artificial Intelligence (LNAI), D. Weyns, S. Brueckner, and Y. Demazeau, Eds., no. 5049. http://www.springerlink.com/: Springer- Verlag, juillet 2008, p. (on line). [Online]. Available: http://www.springerlink.com/content/j62761544x357251/ [accessed: September 7, 2009]

[8] A. Meisels, E. Kaplansky, I. Razgon, and R. Zivan, “Comparing performance of distributed constraints processing algorithms,” in AAMAS-02 Workshop on Distributed Constraint Reasoning, no. 5, 2002, pp. 86–93.

[9] G. Clair, E. Kaddoum, M.-P. Gleizes, and G. Picard, “Self-Regulation in Self-Organising Multi-Agent Systems for Adaptive and Intelligent Manufacturing Control,” in IEEE International Conference on Self-Adaptive and Self-Organizing Systems (SASO), Venice Italy, 20/10/08-24/10/08. http://www.computer.org: IEEE Computer Society, 2008, p. (on line). [Online]. Available: http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=4663415&isnumber=4663391 [accessed: September 7, 2009]

[10] K. M. Hansen, W. Zhang, and M. Ingstrup, “Towards selfmanaged executable petri nets,” in SASO ’08: Proceedings of the 2008 Second IEEE International Conference on Self- Adaptive and Self-

144

Organizing Systems. Washington, DC, USA: IEEE Computer Society, 2008, pp. 287–296.

[11] D. Meignan, J.-C. Creput, and A. Koukam, “A coalition-based metaheuristic for the vehicle routing problem,” June 2008, pp. 1176–1182.

[12] J. J. Dongarra, “Performance of various computers using standard linear equations software in a fortran environment,” SIGARCH Comput. Archit. News, vol. 16, no. 1, pp. 47–69,1988.

[13] N. Lynch, Distributed Algorithms. Morgan-Kaufmann, 1996. [14] I. Brito, F. Herrero, and P. Meseguer, “On the evaluationof discsp

algorithms,” The Fifth International Workshop on Distributed Constraint Reasoning (DCR04) Toronto, Canada September 27, 2004 held in conjunction with Tenth International Conference on Principles and Practice of Constraint Programming (CP 2004), no. 5, pp. 142–151, 2004.

[15] H. Parunak and S. Brueckner, “Engineering swarming systems,” Methodologies and Software Engineering for Agent Systems, pp. 341–376, 2004.

[16] F. Gaillard, Y. Kubera, P. Mathieu, and S. Picault, “A reverse engineering form for multi agent systems,” in Proceedings of the 9th International Workshop Engineering Societies in the Agents World (ESAW’2008), A. Artikis, G. Picard, and L. Vercouter, Eds., 2008, actes lectroniques uniquement. [Online]. Available: http://www.emse.fr/esaw08/technical_programme.php [accessed: September 7, 2009]

[17] M. Chiarandini, “Stochastic local search methods for highly constrained combinatorial optimisation problems,” Ph.D. dissertation, Udine, Italy, 2005.

[18] L. Fortnow and S. Homer, “A short history of computational complexity,” in The History of Mathematical Logic. North- Holland, 2003.

[19] H. S. Wilf, Algorithms and Complexity. Upper Saddle River, NJ 07458, USA: Prentice-Hall, 1986.

[20] C. H. Papadimitriou, A. A. Sch¨affer, and M. Yannakakis, “On the complexity of local search,” in STOC ’90: Proceedings of the twenty-second annual ACM symposium on Theory of computing. New York, NY, USA: ACM, 1990, pp. 438–445.

[21] M. Jelasity and O. Babaoglu, “T-Man: Gossip-Based Overlay Topology Management,” in Engineering Self-Organising Systems: Third International Workshop, ESOA 2005, Utrecht, The Netherlands, July 25, 2005, Revised Selected Papers, ser. Lecture Notes in Computer Science, S. A. Brueckner, G. Di Marzo Serugendo, D. Hales, and F. Zambonelli, Eds. Springer-Verlag, 2006, pp. 1–15.

[22] S. Minton, M. Johnston, A. Philips, and P. Laird, “Minimizing Conflicts: a Heuristic Repair Method for Constraint Satisfaction and Scheduling Problems,” Constraint-Based Reasoning, vol. 58, no. 1-3, pp. 161–205, 1994.

[23] J. Dro, A. Ptrowski, P. Siarry, and E. Taillard, Metaheuristics for Hard Optimization: Methods and Case Studies. Springer, 2005.

[24] G. Bonnet and C. Tessier, “Multi-agent collaboration: A satellite constellation case.” in STAIRS, ser. Frontiers in Artificial Intelligence and Applications, A. Cesta and N. Fakotakis, Eds., vol. 179. IOS Press, 2008, pp. 24– 35.

[25] K. Hirayama and M. Yokoo, “The distributed breakout algorithms,” Artificial Intelligence, vol. 161, no. 1-2, pp. 89–115, 2005.

[26] J. Liu, H. Jing, and Y. Y. Tang, “Multi-agent Oriented Constraint Satisfaction,” Artificial Intelligence, vol. 136, no. 1, pp. 101–144, 2002.

[27] M. Dorigo and T. St¨utzle, Ant Colony Optimization. MIT Press, 2004.

[28] J. Kennedy and R. C. Eberhart, Swarm Intelligence. Morgan Kaufmann, 2001.

[29] G. Picard and P. Glize, “Model and analysis of local decision based on cooperative self-organization for problem solving,” Multiagent and Grid Systems (MAGS), vol. 2, no. 3, pp. 253– 265, 2006.

[30] A. Charles, R. Menezes, and R. Tolksdorf, “On the implementation of swarmlinda,” in Proceedings of the 42nd Annual Southeast Regional Conference (ACM-SE). ACM Press, 2004, pp. 297–298.

[31] M. Mamei and F. Zambonelli, “Programming Stigmergic Coordination with the TOTA Middleware,” in Proceedings of the 4th International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS 2005). ACM, 2005, pp. 415–422.

[32] W. Weber, J. Rabaey, and E. Aarts, Ambient Intelligence. Springer, 2005.

[33] P. De Loor and P. Chevaillier, “Solving distributed and dynamic contraints using an emotional metaphor : Application to the timetabling problem,” in 5th EURO/INFORM international conference, new opportunities for operations research, Jul. 2003.

[34] http://jade.tilab.com/ [accessed: September 7, 2009] [35] http://www.irit.fr/MAY [accessed: September 7, 2009]

145