138
AgentSpotter: a MAS Profiling System for Agent Factory Dinh Doan Van Bien Thesis presented for the degree of Masters in Advanced Software Engineering to the School of Computer Science & Informatics College of Engineering, Mathematical & Physical Sciences University College Dublin Research Supervisor: Dr. Rem W. Collier Head of School: Dr. Joe Carthy October 2008

AgentSpotter: a MAS Profiling System for Agent Factory

Embed Size (px)

DESCRIPTION

Recent advances in Agent Oriented Software Engineering (AOSE)have focused on providing agent-oriented frameworks and integratedtools in order to facilitate the creation of multiagent systems (MAS).The inherent complexity of MASs means that performance and scala-bility issues are likely to arise. Yet there is a lack of industry-gradetools that enable software engineers to understand and debug thesecrucial problems.In this dissertation, we address this deciency through the devel-opment of a novel MAS performance analysis system entitled Agent-Spotter, based on the mapping of conventional proling conceptsto agent-oriented concepts. AgentSpotter's platform-independentarchitecture is composed of a MAS run-time proling data collec-tion infrastructure and a suite of interactive visualisation tools (FlatProle, Call Graph Tree View, and Space Time Diagram) that usethe collected data to provide new insights into a MAS run-timeperformance.We present our implementation of AgentSpotter for Agent Factory(AF), a Java-based agent-oriented toolkit developed by researchersat the PRISM Laboratory (University College Dublin).Finally, we demonstrate the eectiveness of AgentSpotter byusing it to prole a set of representative MASs developed with AgentFactory.

Citation preview

Page 1: AgentSpotter: a MAS Profiling System for Agent Factory

AgentSpotter: a MAS Profiling System for

Agent Factory

Dinh Doan Van Bien

Thesis presented for the degree of

Masters in Advanced Software Engineering

to the

School of Computer Science & Informatics

College of Engineering, Mathematical & Physical Sciences

University College Dublin

Research Supervisor: Dr. Rem W. Collier

Head of School: Dr. Joe Carthy

October 2008

Page 2: AgentSpotter: a MAS Profiling System for Agent Factory
Page 3: AgentSpotter: a MAS Profiling System for Agent Factory

Abstract

Recent advances in Agent Oriented Software Engineering (AOSE)

have focused on providing agent-oriented frameworks and integrated

tools in order to facilitate the creation of multiagent systems (MAS).

The inherent complexity of MASs means that performance and scala-

bility issues are likely to arise. Yet there is a lack of industry-grade

tools that enable software engineers to understand and debug these

crucial problems.

In this dissertation, we address this deficiency through the devel-

opment of a novel MAS performance analysis system entitled Agent-

Spotter, based on the mapping of conventional profiling concepts

to agent-oriented concepts. AgentSpotter’s platform-independent

architecture is composed of a MAS run-time profiling data collec-

tion infrastructure and a suite of interactive visualisation tools (Flat

Profile, Call Graph Tree View, and Space Time Diagram) that use

the collected data to provide new insights into a MAS run-time

performance.

We present our implementation of AgentSpotter for Agent Factory

(AF), a Java-based agent-oriented toolkit developed by researchers

at the PRISM Laboratory (University College Dublin).

Finally, we demonstrate the effectiveness of AgentSpotter by

using it to profile a set of representative MASs developed with Agent

Factory.

Page 4: AgentSpotter: a MAS Profiling System for Agent Factory
Page 5: AgentSpotter: a MAS Profiling System for Agent Factory

Contents

Abstract iii

Contents v

List of Figures ix

List of Tables xi

Acknowledgements xiii

1 Introduction 1

1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.3 Thesis outline . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Agents and multiagent systems 5

2.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.2 FIPA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.3 Agent Factory . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.4 JADE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.5 Cougaar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3 Profiling 9

v

Page 6: AgentSpotter: a MAS Profiling System for Agent Factory

vi Contents

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.3 Origins of profiling . . . . . . . . . . . . . . . . . . . . . . . 11

3.4 First generation profilers . . . . . . . . . . . . . . . . . . . . 13

3.5 Second generation profilers . . . . . . . . . . . . . . . . . . . 14

3.6 Modern Java profilers . . . . . . . . . . . . . . . . . . . . . . 15

3.7 Profiling in perspective . . . . . . . . . . . . . . . . . . . . . 16

3.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

4 Performance concepts and solutions in MASs 19

4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.2 Fundamental performance issues in a MAS context . . . . . 21

4.3 MAS scalability issues . . . . . . . . . . . . . . . . . . . . . 23

4.4 Facilitating MAS performance case studies . . . . . . . . . . 27

4.5 Cougaar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

4.7 Towards a set of requirements for a performance analysis toolkit 34

5 A blueprint for AgentSpotter, the Agent Factory profiling

system 37

5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

5.2 Platform-independent AgentSpotter architecture . . . . . . . 39

5.3 Platform-specific AgentSpotter architecture for Agent Factory 43

5.4 AgentSpotter Station . . . . . . . . . . . . . . . . . . . . . . 45

5.5 Session Summary Tables . . . . . . . . . . . . . . . . . . . . 46

5.6 Call Graph Tree View . . . . . . . . . . . . . . . . . . . . . 49

5.7 Space-Time Diagram concept . . . . . . . . . . . . . . . . . 56

5.8 Space-Time Diagram user interface . . . . . . . . . . . . . . 60

Page 7: AgentSpotter: a MAS Profiling System for Agent Factory

Contents vii

5.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

6 AgentSpotter design and implementation 69

6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

6.2 AgentSpotter implementation architecture: an overview . . . 70

6.3 AgentSpotter profiling infrastructure . . . . . . . . . . . . . 72

6.4 AgentSpotter File Manager . . . . . . . . . . . . . . . . . . 75

6.5 AgentSpotter Station . . . . . . . . . . . . . . . . . . . . . . 76

6.6 AgentSpotter Visualisation Components . . . . . . . . . . . 77

6.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

7 Agent Spotting: an evaluation 85

7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

7.2 Benchmark application . . . . . . . . . . . . . . . . . . . . . 86

7.3 Agent Sherpa . . . . . . . . . . . . . . . . . . . . . . . . . . 90

7.4 Agent Herders . . . . . . . . . . . . . . . . . . . . . . . . . . 97

7.5 Evaluation summary . . . . . . . . . . . . . . . . . . . . . . 98

8 Conclusions 103

8.1 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . 103

8.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

8.3 General conclusion . . . . . . . . . . . . . . . . . . . . . . . 105

Appendices 109

A Building AgentSpotter from the subversion repository 109

B Getting started with AgentSpotter 111

B.1 Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

B.2 AgentSpotter Station . . . . . . . . . . . . . . . . . . . . . . 112

Page 8: AgentSpotter: a MAS Profiling System for Agent Factory

viii Contents

C AgentSpotter Snapshot File Database Schema 115

Bibliography 117

Page 9: AgentSpotter: a MAS Profiling System for Agent Factory

List of Figures

3.1 Profilers time line . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5.1 AgentSpotter abstract (platform-independent) architecture . . . 40

5.2 AgentSpotter for Agent Factory concrete architecture . . . . . . 40

5.3 Call Graph Tree View of a fictional C program which removes

duplicate lines from a file . . . . . . . . . . . . . . . . . . . . . . 49

5.4 Agent message impact concept diagram . . . . . . . . . . . . . . 51

5.5 AgentSpotter Call Graph Tree View levels . . . . . . . . . . . . 53

5.6 Lamport Space-Time Diagram . . . . . . . . . . . . . . . . . . . 56

5.7 AgentSpotter Space-Time Diagram specification . . . . . . . . . 59

5.8 Agent Factory agent activity representation in the Space-Time

Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

6.1 AgentSpotter implementation architecture . . . . . . . . . . . . 71

6.2 Agent Spotter Station sample screen shot showing all the available

views in a single take . . . . . . . . . . . . . . . . . . . . . . . . 78

6.3 Call Graph Tree View screen shot . . . . . . . . . . . . . . . . . 79

6.4 Space-Time Diagram screen shot . . . . . . . . . . . . . . . . . 84

7.1 Benchmark application control window . . . . . . . . . . . . . . 90

ix

Page 10: AgentSpotter: a MAS Profiling System for Agent Factory

x List of Figures

7.2 Benchmark application sample space-time diagram (18 minute

long session) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

7.3 Comparison of Agent Sherpa call graphs . . . . . . . . . . . . . 95

7.4 Comparison of Agent Sherpa space-time diagrams . . . . . . . . 96

7.5 Agent Herders’ call graph . . . . . . . . . . . . . . . . . . . . . 100

A.1 Netbeans “Open Project” dialog box . . . . . . . . . . . . . . . 110

C.1 Database schema diagram of the AgentSpotter profile data file 115

Page 11: AgentSpotter: a MAS Profiling System for Agent Factory

List of Tables

4.1 Common MAS scalability problems and solutions . . . . . . . . 25

5.1 OOP and AOP execution units mapping . . . . . . . . . . . . . 47

6.1 TraceEvent types generated by the AF-RTE sub-systems . . . . 73

7.1 Benchmark application flat profile . . . . . . . . . . . . . . . . . 91

7.2 Agent Sherpa application flat profile for 3 drone session . . . . . 94

7.3 Agent Sherpa application flat profile for 12 drone session . . . . 97

7.4 Agent Herders’ flat profile . . . . . . . . . . . . . . . . . . . . . 99

7.5 Comparison of the profiled sessions in terms of number of events 99

7.6 Comparison of AgentSpotter visualisations load times on a 2GHz

Dual Core Pentium machine . . . . . . . . . . . . . . . . . . . . 101

xi

Page 12: AgentSpotter: a MAS Profiling System for Agent Factory
Page 13: AgentSpotter: a MAS Profiling System for Agent Factory

Acknowledgements

I would like to thank my thesis supervisor Rem W. Collier for his outstanding

support and guidance.

My thanks also go to my company, Intelligo Software Ltd, for sponsoring

my Master’s degree.

Finally, I am deeply grateful to my wife Stephanie for everything she has

done to make it all happen.

xiii

Page 14: AgentSpotter: a MAS Profiling System for Agent Factory
Page 15: AgentSpotter: a MAS Profiling System for Agent Factory

CHAPTER 1Introduction

1.1 Motivation

Recent advances in agent frameworks and methodologies have helped bridg-

ing the gap between agent-oriented programming theory and practice. Yet

there is still a lot of work to do before agent-oriented software engineering

(AOSE) can truly break into industry, notably in the area of development

and debugging tools.

To this end, agent-oriented frameworks have been integrated into existing

mainstream Java development environments (IDEs) like NetBeans or Eclipse

to allow programmers to use existing tools and techniques originally designed

for an object-oriented context.

This reuse can work well at the source code level or the agent code level,

because the required activities’ nature and granularity are quite similar:

syntax highlighting, project building, stepping through the code. In other

words, the ancillary tasks of project and build management are conceptually

the same in object-oriented and agent-oriented software engineering, and

can be performed using the same tools and techniques. For example, in

1

Page 16: AgentSpotter: a MAS Profiling System for Agent Factory

2 Introduction

Agent Factory (AF) [11], a Java-based agent-oriented toolkit, a project is

constituted of a set of source code files, written in Java or in AFAPL1, plus

a set of configuration and deployment files. An AF project can be built into

an executable set of JAR files using the standard Java build tool Ant.

At run-time, the analogy between object-oriented and agent-oriented can

be pushed further. As long as the object-oriented level is the appropriate

abstraction level for the problem at hand, an agent-oriented programmer

can efficiently use all the run-time debugging, memory inspection, and

performance analysis services provided by modern IDEs.

In fact, the AOSE proponents generally agree that the full integration

with existing IDEs is an important step towards the widespread adoption

of the agent-oriented paradigm, because it allows novice agent-oriented

programmers to build on their existing object-oriented skills in order to

create multiagent systems (MASs).

Nevertheless, we argue that modern object-oriented IDEs do not cover

all the needs of agent-oriented programmers “out of the box”, notably when

they are dealing with issues at the higher agent-oriented abstraction level.

Despite the recent development of AOSE tools like the AF Debugger

tool [7], which maps the traditional debugging concepts of state stepping

and breakpoint setting onto intelligent agents, tools that target software

developers whilst providing new insights into agent-oriented systems are

scarce. Moreover, after surveying the literature, we have determined that

one area is particularly lacking: MAS performance analysis tools.

It is commonly admitted that the behaviour of complex and distributed

MASs is very difficult to apprehend without adequate tools. To address this

problem, existing MAS toolkits usually offer a more or less advanced logging

1Agent Factory Agent Programming Language

Page 17: AgentSpotter: a MAS Profiling System for Agent Factory

1.2. Objectives 3

system to help developers debug and understand their systems [10, 19].

While logging can deal with simple scenarios, software developers expect

more advanced tools to tackle more elusive and complex performance issues.

Ideally, MAS developers should be able to use a very popular performance

analysis technique called profiling. Profiling is based on the idea that in a

program, only a few places, called bottlenecks or hot spots, can be accounted

for the majority of the execution time of a program. Hence, by fixing

these bottlenecks only, the performance of a program can be substantially

improved. Profiling was introduced 40 years ago by Donald E. Knuth

in his empirical study of FORTRAN programs [26], and has since then

been successfully adapted to different languages, platforms and software

architectures, including large distributed systems.

We strongly believe that a profiler could truly help MAS developers,

novice or experimented. Yet, as far as we know, profiling concepts have

never been applied to agent-oriented programming. Therefore, we propose

to implement the first profiling system for a MAS.

1.2 Objectives

Our main objectives are as follows:

1. To map profiling concepts to agent-oriented concepts.

2. To elaborate a conceptual model for generic MAS profiling.

3. To realise and evaluate a profiling system based on this conceptual

model for a specific agent platform, namely Agent Factory.

In short, we aim to build a comprehensive MAS performance analysis

system for Agent Factory, based on profiling.

Page 18: AgentSpotter: a MAS Profiling System for Agent Factory

4 Introduction

1.3 Thesis outline

We have classically organised this dissertation in two main parts:

1. Fundamental Concepts and Background Research, which includes the

following chapters:

• Chapter 2 – Agents and multiagent systems.

• Chapter 3 – Profiling.

• Chapter 4 – Performance concepts and solutions in MASs.

2. Project Presentation, which includes the following chapters:

• Chapter 5 – A blueprint for AgentSpotter, the Agent Factory

profiling system.

• Chapter 6 – AgentSpotter design and implementation.

• Chapter 7 – Agent Spotting: an evaluation.

Page 19: AgentSpotter: a MAS Profiling System for Agent Factory

CHAPTER 2Agents and multiagent systems

In this chapter, we present the fundamental notions of software agent and

multiagent system. We also briefly provide background information about

the different systems that we will reference later.

2.1 Definitions

For our purposes, a multiagent system (MAS) is a software system, situated

in a given environment, and composed of interacting coarse grained software

entities called agents. The most advanced agents are capable of intelligent

goal-directed autonomous behaviour (deliberative agents), whereas less

capable agents can only react to external events (reactive agents), but still

retain their autonomy by controlling their own thread of execution. Agents

have no direct control over their environment, instead they sense it through

sensors and act upon it using actuators. The interaction between agents

happens through an agent communication language which allows an agent

to influence the behaviour of another agent by sending messages based, for

instance, on the theory of speech acts1.1See the FIPA ACL message structure specification in [15].

5

Page 20: AgentSpotter: a MAS Profiling System for Agent Factory

6 Agents and multiagent systems

In the perspective of the two-tier properties-based approach to agent

definition proposed by Wooldridge and Jennings in [52], our definition fully

covers the weak notion of agency (autonomy, social ability, reactivity, pro-

activeness), and relies on the stronger agency concepts of rationality and

intentionality.

Recent advances in MAS research have led to the emergence of agent-

oriented software frameworks, mostly Java-based, that allow the development

and the deployment of multiagent systems using standard software infras-

tructures and development tools. These frameworks promote the concepts of

agent-oriented programming (AOP) and agent-oriented software engineering

(AOSE).

Our work is based on the Agent Factory (AF) framework, described later

in §2.3. So, by all means, our working definition of an agent matches the

one underlying AF.

2.2 FIPA

The Foundation for Intelligent Physical Agents (FIPA) is an organisation

created in 1996 in Switzerland to lead the standardisation effort in agent-

based technologies [23]. Since 2005, FIPA has been officially accepted by

the IEEE as its eleventh standards committee. FIPA addresses a wide range

of agent-oriented technologies. The most important standards are surely the

Abstract Architecture specification [16] and the FIPA Agent Communication

Language (ACL) specification [15]. FIPA standards are widely adopted and

guarantee a minimum level of interoperability between the different agent

platforms.

Page 21: AgentSpotter: a MAS Profiling System for Agent Factory

2.3. Agent Factory 7

2.3 Agent Factory

AF is an open source Java-based, comprehensive toolkit for the creation

and deployment of agent-oriented applications. It is actively maintained by

researchers at the PRISM Laboratory in the School of Computer Science

and Informatics at University College Dublin. It has successfully supported

advanced research in a diverse range of problem domains: mobile & ubiqui-

tous computing, distributed sensor networks, social robotics, and augmented

reality [8].

It is structured around a FIPA-compliant run-time environment, which

offers basic system services (agent management, messaging, yellow pages),

and a system of pluggable agent development kits. The default AFAPL2

development kit provides a language and a reasoning engine for BDI (Belief

Desire Intention) agents, but other agent models can be plugged in like

the Reactive Message Agent model for example (RMA development kit).

Beyond its integration with the Java subsystem, a strong point of Agent

Factory is its integration with the mainstream Java IDEs NetBeans and

Eclipse.

2.4 JADE

JADE stands for “Java Agent DEvelopment Framework”. It is a major

open source project that benefits from an industrial level support through

the http://jade.tilab.com web site [50]. An extensive set of academic

papers, tutorials, and programming guides is available on-line, and recently

a comprehensive book geared towards developers was published [2]. JADE

has been used in a wide range of applications by companies and academic

bodies. It is so popular that on their web site the JADE maintainers claim

that “its spread surpasses even JADE team knowledge” and that they

Page 22: AgentSpotter: a MAS Profiling System for Agent Factory

8 Agents and multiagent systems

“apologize in advance for all those [companies or institutions] that have not

been listed” [50].

2.5 Cougaar

Cougaar is an open source Java-based platform particularly suited to the

construction of large scale MASs. Cougaar was originally funded by the

American Army (DARPA) and developed for military grade logistics deploy-

ment planning applications. In 2001 it was transferred to the commercial

domain as an “intelligent middleware” and has since then benefited from an

active commercial support. Cougaar is the only existing MAS framework

which integrates a performance measurement infrastructure right into the

system architecture.

Page 23: AgentSpotter: a MAS Profiling System for Agent Factory

CHAPTER 3Profiling

3.1 Introduction

In this chapter, we describe profiling and explain why it matters in software

engineering. Our objective is to provide a solid understanding of profiling

concepts before we try to apply them to agent-oriented programming.

To this end, we start by defining what profiling is. Then, we explore

the elaboration of profiling concepts throughout history. Finally, we place

profiling in the greater perspective of software engineering.

3.2 Definitions

Most of the time in the software engineering literature, the definition of

profiling delves directly into technical aspects such as differentiating in-

struction counting profilers from sampling profilers, and it is assumed that

its underlying principles are well known and obvious. We believe a useful

definition should instead focus on these principles in order to understand

why profiling is so useful and so popular . Hence, we propose our own

9

Page 24: AgentSpotter: a MAS Profiling System for Agent Factory

10 Profiling

definition of profiling which tries to incorporate the elements that make

profiling an effective performance analysis technique.

General Definition. We define profiling as a performance analysis process

which consists in collecting, processing and analysing run-time data about

where a program spends its time, or how it spends its resources, in order

to gain a fresh insight into the main factors of the program’s performance,

or to detect hidden behavioural and resource usage patterns. The primary

motivation for profiling is performance debugging or tuning, which involves

iteratively identifying and removing the hot spots (areas that use much more

processing time than expected) or bottlenecks (areas that use processing time

or resources inefficiently and introduce unnecessary delays) in a program

until a satisfying performance level is reached.

Profiling involves the use of a profiler, which is composed of two equally

important logical parts:

• An instrumentation apparatus which is directly weaved into the pro-

gram under study or run side-by-side to gather and record execution

data;

• A post-processing system which uses the recorded data to generate

meaningful performance analysis listings or visualisations.

Limited Definition. A more limited definition of profiling designates only the

process of collecting aggregated performance data over the whole execution

of a program. The data generated this way is called a profile and is

constituted of counters, aggregated timings, or summaries. A profile must be

distinguished from a trace, which records the actual chronology of execution

events and generates a growing amount of data over run-time.

Page 25: AgentSpotter: a MAS Profiling System for Agent Factory

3.3. Origins of profiling 11

The limited definition does not contradict our general definition, but

it fails to emphasise the key value of profiling which is the eye-opening

quality of the information it extracts from raw performance data, providing

real insight into what drives or hampers the performance of a program and

revealing hidden patterns.

As explained in the next section, that eye-opening quality is what has

made profiling so remarkable right from its inception.

In the following sections, we summarise the history of profiling, following

the time line presented in Figure 3.1.

3.3 Origins of profiling

Nearly forty years ago, in his 1971 paper on the empirical study of FORTRAN

programs [26], Donald E. Knuth introduced a simple yet powerful dynamic

program statistical analysis technique that is still widely in use today:

program profiles.

Originally, a program profile or “frequency counts” was an annotated

program listing showing for each statement the number of times it was

actually executed and its estimated cumulative cost in terms of machine

cycles.

The profiles Knuth generated for his FORTRAN sample programs re-

vealed that “less than 4% of a program generally accounts for more than

half of its running time” [26], which implies that a programmer can make

substantial improvements to a program by using the program’s profile to

identify and remove the hot spots or bottlenecks that account for the ma-

jority of the execution time. This principle is sometimes called the “Pareto

Principle” or the “80/20 law”; it has been universally observed in domains

other than software development [40], and stated as the “vital few and trivial

Page 26: AgentSpotter: a MAS Profiling System for Agent Factory

12 Profiling

1970 1980 1990 2000 2010

Knuth

Pro lers

First profilers

gprof

Call graph executionprofiler

ATOM

Customized programanalysis tools

JVMPI

First Java profilerinterface API

JVMTI

New tool interfaceAPI, JVMPI replacement

prof

Unix profiler

Figure 3.1: Profilers time line

many” [48] — meaning that in anything a few are vital (20%), and many

are trivial (80%).

Interestingly, Knuth also discovered that often the programmers’ mental

map of the bottlenecks in their programs did not match the actual findings

of program profiles. In other words, profiles were real “eye-openers” and

gave them an unexpected insight into the actual run-time behaviour of their

programs (as opposed to the predicted behaviour). That fact advocates the

necessity of using tools that can help the programmer break the limitations

of their minds and analyse information about running software, which

is otherwise intractable because of the sheer amount of data and events

produced over run-time.

In retrospect, Knuth’s findings have been very influential and have

implicitly driven most of the performance analysis approaches in software

development in the last three decades.

Page 27: AgentSpotter: a MAS Profiling System for Agent Factory

3.4. First generation profilers 13

3.4 First generation profilers

3.4.1 Knuth profilers

Technically, the first profilers were the programs that Knuth used for his 1971

study [26]: FORDAP, based on instruction counting, and PROGTIME, based

on program status sampling. Their underlying principles were applied with

nearly no modifications to the first generation of profiling tools represented

by prof and pxp.

FORDAP would take a FORTRAN program as input and would output

an equivalent program that would also maintain frequency counts and write

them into a file. After being compiled and run, the modified program would

produce an annotated listing showing for each statement the number of

times it was actually performed and the estimated total execution time.

PROGTIME was based on a different approach called program status

sampling. PROGTIME would spawn the user program as a subtask and

stop at regular intervals to record with a time stamp the place where the

program under observation was executing. Instead of frequency counts,

PROGTIME would produce execution time listings.

Theoretically both approaches should reveal the same bottlenecks. How-

ever, program status sampling gives a more realistic view of the run-time

behaviour of a program by showing more clearly the low-frequency but time

consuming operations like input and output.

3.4.2 UNIX prof

In 1979, the UNIX implementers introduced a sampling profiler for the C

language called prof.

The main output of prof is a “flat profile” which shows how many times

a program spends in each function, and how many times that function is

Page 28: AgentSpotter: a MAS Profiling System for Agent Factory

14 Profiling

called. The sampling method used by prof means that it is less accurate

than an instruction counting profiler, also called “exact” profiler. When

the execution time of a function is smaller than the sampling interval, it

won’t be properly recorded. Despite these shortcomings, prof is reasonably

accurate for most uses.

Its use involves several steps:

• Compile the program using cc with the “-p” switch to instrument the

program.

• Run the program. After completion it generates a “mon.out” file

which contains run-time statistics.

• Set the current shell directory to the “mon.out” file location.

• View the statistics by typing “prof” on the command line.

prof is the most widely available profiler on UNIX systems, and is indeed

so popular that it is still in use today.

3.5 Second generation profilers

3.5.1 gprof: a call graph execution profiler

The main contribution of gprof, introduced in the 80’s by Graham et al. [17],

is the call graph. This concept improves on prof’s flat profile. It not only

summarises the time spent in different functions, but also recursively presents

all the call stacks annotated with the time spent in the different functions

called. Another name for the call graph is “hierarchical profile”, which

conveys the idea that gprof provides information to understand the impact

of one function in relation with all the functions that have called it. To put

Page 29: AgentSpotter: a MAS Profiling System for Agent Factory

3.6. Modern Java profilers 15

it crudely, a call graph shows the “where” to help understand why the “how

long” is a problem.

The textual output of gprof is very dense and requires some practice to

understand. We shall see in §3.6 how modern profiler user interfaces have

made the call graph more tractable.

3.5.2 ATOM

This instrumentation framework, created in the 90’s by Srivastava and

Eustace [47], has popularised the concept of allowing software engineers

to create custom program analysis tools. In short, ATOM provides an

infrastructure and a simple API which enables the creation of different tools

without having to start from scratch. ATOM has been used to create profilers,

execution tracers, run-time optimisers etc. That is to say, programmers who

are not satisfied with the service offered by standard tools can relatively

easily implement a tool tailored to their needs. This idea is the basis of

modern program analysis APIs as we shall see in the next section.

3.6 Modern Java profilers

The Java Virtual Machine Tool Interface (JVMTI) [49] and assorted tools

inherit from the systems and concept we have previously mentioned. The

main motivation of JVMTI is to provide a standard API like ATOM both

for specialised tool vendors and for casual developers with punctual instru-

mentation needs. The API gives access to JVM internal information items

(call stacks, class metadata, CPU usage counters etc.) that can be pieced

together to create a monitoring application, a profiler or even a debugger.

Modern Java profilers use the JVMTI as their back-end, and offer ad-

vanced technical features like “on-demand” profiling, remote profiling, thread

profiling etc. However, from a conceptual point of view, modern profilers

Page 30: AgentSpotter: a MAS Profiling System for Agent Factory

16 Profiling

have not introduced new ideas, they have just improved on the existing

concepts we presented: flat profiles, call graphs, profiling API. To be fair,

the adoption of graphical user interfaces has made profilers more usable

than ever. For instance, the visual representation of call graphs in tree view

controls has made their interpretation much easier.

3.7 Profiling in perspective

Performance is an important attribute of good software. Efficient software

should not make wasteful use of system resources, and should be as responsive

as expected by its users.

Having said that, from a business point of view it does not make sense to

be overly concerned by optimal performance when designing software, unless

it either poses a genuine problem (e.g. a program is not responsive enough

to satisfy its users or overruns its resources limitations), or is a primary

system goal per se (as in real time software).

That explains why the majority of programmers choose to use perfor-

mance analysis tools only punctually, when confronted with severe perfor-

mance problems. In other words, programmers will favour easy to use tools

that produce a substantial amount of improvements with the minimum

amount of effort.

3.8 Conclusion

In summary, we defined profiling and explained why it matters for program-

mers. We also retraced the history of profiling and concluded that modern

profilers are still using concepts invented one or two decades ago. All things

considered, programmers have a predilection for profiling when conducting

performance analysis, because it has a high benefit to effort ratio.

Page 31: AgentSpotter: a MAS Profiling System for Agent Factory

3.8. Conclusion 17

Clearly, profiling could be very useful for MAS developers. So how should

we proceed to map profiling concepts to agent-oriented programming1?

In a traditional OOP context, profiling is focused on methods, and

restricted to program or JVM boundaries, with the primary objective of

detecting bottlenecks by looking at method calls frequency counts, method

execution timings, call graphs, memory usage graphs etc.

In a MAS context, however, the profiles and the associated visualisations

are generally too fine-grained and too low level for a relevant analysis of

MAS run-time performance and activities. We shall see in Chapter 5 how

these concepts can be mapped to a MAS context. Beforehand, we need to

provide more theoretical background about the performance issues in MASs.

1We must mention that profiling has been successfully ported to different programmingparadigms. We think the most striking example is the work of Brassel et al., who describein [3] how they have applied run-time profiling to functional logic programs.

Page 32: AgentSpotter: a MAS Profiling System for Agent Factory
Page 33: AgentSpotter: a MAS Profiling System for Agent Factory

CHAPTER 4Performance concepts and

solutions in MASs

4.1 Introduction

In the light of the history and the current state of performance analysis

tools in non agent-oriented systems, and with the aim of developing a MAS

performance analysis system, the next logical step is to proceed to a survey of

the existing work on MAS performance, with a particular focus on integrated

tool-based solutions.

A first glance at the literature reveals a contrasted picture. On the one

hand, there is a good amount of literature on scalability, which is commonly

admitted as the most significant performance related concept in multiagent

systems. The main motivation of these studies is to provide solutions to the

scalability issues of potentially large MASs that enterprise level applications

require. The solutions described borrow heavily from the distributed systems

research area [13, 4] while trying to come up with concepts truly unique

to the agent-oriented paradigm like self-organising [51, 38, 6, 30] and self-

19

Page 34: AgentSpotter: a MAS Profiling System for Agent Factory

20 Performance concepts and solutions in MASs

tuning systems [30, 6]. On the other hand, it is harder to find papers

about comprehensive solutions that try to reduce the gap between theory

and practice and help non-expert, or impatient, MAS designers conduct

meaningful performance analyses of their system under construction. The

only example of an integrated performance analysis infrastructure that we

found is the Cougaar system described in [19].

Our point is that although MAS performance, specifically scalability, is a

seemingly well studied subject, there is still some work to do to translate the

general conclusions into a comprehensive and usable performance analysis

toolkit integrated with a modern agent platform, which we insist on as being

an important requirement to fulfil if we want to allow non-specialist software

engineers to design enterprise level MASs. Indeed, a growing number of

MASs are leaving the experimental arena for the production environments,

and given the more stringent quality of service requirements of enterprise-

level production systems and the likeliness of emergent complex behaviour to

exhibit performance issues, it is obvious that performance analysis will be an

important area in AOSE even if performance is not a primary requirement,

as in parallel programming for instance.

What’s more, the software engineers are much wiser now and will not put

up with reams of logging statements or time-consuming wheel-reinventing

experiments anymore. Even when using the modest JavaScript browser

language, they now are just a mouse click away from a wide choice of

profilers and other powerful performance analysis tools that provide detailed

performance reports and summarised performance indicators [37], and can

even suggest performance improvements automatically [53].

As a result we chose to structure our survey in a way that provides

a strong foundation for a practical approach to MAS performance anal-

Page 35: AgentSpotter: a MAS Profiling System for Agent Factory

4.2. Fundamental performance issues in a MAS context 21

ysis. Rather than simply enumerating the existing approaches to MAS

performance in a thematic fashion, our survey is organised around a prob-

lem/solution discussion that helps us progressively motivate the creation of

a MAS performance analysis toolkit.

In the first part of this chapter, we present a study of the fundamental

performance issues in MAS context and identify the major approaches

promoted in the literature to address these issues.

In the second section of this chapter, we show how MAS performance

analysis case studies could benefit from the existence of an integrated

performance analysis system.

In the third part of this chapter, we describe an existing performance

measurement infrastructure that could be used as the basis for a performance

analysis system.

4.2 Fundamental performance issues in a MAS context

Our goal is to present a theoretical view of the main issues of MAS per-

formance, and to identify the main solutions that have been proposed to

address these issues. We start by a necessary characterisation of the perfor-

mance of agent-oriented software and a definition of essential concepts like

performance and scalability. Then we proceed to a more detailed analysis

of scalability which is commonly viewed as the central performance related

issue in MASs. Finally we expose the different approaches proposed to

address scalability issues in MASs.

4.2.1 Basic performance concepts

In [45], Smith and Williams define performance as being a measure of how

quickly a software system or component executes compared to some prede-

fined time objectives. In other words, performance designates any aspect of

Page 36: AgentSpotter: a MAS Profiling System for Agent Factory

22 Performance concepts and solutions in MASs

a software system that you can measure using a stopwatch. The authors

also state that the most important performance concepts are responsiveness

and scalability, and propose the following basic definitions:

Responsiveness measures how well a software system’s response time or

throughput meets a user’s performance requirements.

Response time measures how much time a software system takes to fulfil

an end-user request.

Throughput quantifies the number of requests that can be handled in a

given time slice, usually a minute.

Scalability indicates how well a system can maintain its response time or

its throughput under its performance objectives despite an increasing

workload. Actually, scalability is not a number but a function which

puts in relation the response time with the load, and can be plotted as

a scalability curve. Ideally the scalability curve should grow linearly.

These general definitions provide the basic concepts in order to char-

acterise the performance of software systems. Furthermore, they allow us

to clearly state that the performance of a system is perceived as problem-

atic if the system does not meet its performance objectives in terms of

responsiveness and scalability. To deal properly with performance issues, the

performance objectives that are missed must therefore be clearly identified,

so that the responsiveness and scalability can be fixed and improved.

4.2.2 MAS performance at different levels

In a MAS, performance must be viewed at different levels. At the agent level,

when considering only the autonomous computational entity abstracted

away from the interaction with its peers, the focus is on responsiveness. It

Page 37: AgentSpotter: a MAS Profiling System for Agent Factory

4.3. MAS scalability issues 23

is obvious that the main influence on the responsiveness of an agent is the

amount of computation it requires to carry out the set of tasks required

to meet its design objectives. Agreeing with the BT researchers in [29],

we make a further distinction based on the rationality level of the agent

architecture. For reactive agents this set of tasks includes only purely

reactive behaviour which is similar to the behaviour of a traditional process.

For deliberative agents that implement a reasoning system, the computation

cost of the reasoning activity must be considered separately from the actual

task execution.

As a side note, at the agent platform level, the different platform services

have a running cost that must be accounted for. The main subsystems to

consider include the Agent Management Service and the Message Transport

Service as defined in the FIPA Abstract Architecture Specification [16].

At the risk of stating the obvious, a performance toolkit should clearly

provide access to the essential performance indicators we have defined in

this section.

4.3 MAS scalability issues

At the MAS level, scalability is by far the most studied aspect of agent-

oriented systems performance in the literature, surely because it is commonly

recognised as the main performance issue in distributed systems. We have

summarised in table 4.1 the wide array of scalability issues that can arise

in a MAS. There is not a single approach available to solve these issues,

however the solutions shown in the table can be broadly described as

approaches to manage and reduce the load and the complexity of the system.

These approaches can be further classified depending on their level of agent

orientation. Solutions that propose the use of proved techniques from

Page 38: AgentSpotter: a MAS Profiling System for Agent Factory

24 Performance concepts and solutions in MASs

the distributed systems area can be contrasted with solutions that focus

specifically on agent concepts like improved coordination algorithms, specific

MAS performance models, and at the other end of the scale, solutions that

empower agent systems to deal autonomously with scalability issues.

Of course both points of view are equally valid and interesting, but ulti-

mately we favour the truly novel agent-oriented solutions that demonstrate

the added value of the agent-oriented paradigm.

Additionally, performance case studies of common MAS structures and

architectures constitute an important research direction that can help MAS

designers make more informed design decisions. In the following paragraphs

we provide a more detailed development of each one of these approaches.

4.3.1 Solutions inspired by distributed systems

As explained in [4] and [13], well-known scaling techniques from the dis-

tributed systems area must not be overseen. These techniques include hiding

communication latencies, data and tasks distribution, and data replication.

In practice, Brazier et al. suggest in [4] that MAS designers should base

their implementation of scalable naming or messaging services on existing

middleware architectures or protocols (e.g. CORBA, DNS).

4.3.2 MAS performance prediction and evaluation models

Following the software performance engineering proactive approach [44],

models can be used to predict agent systems performance and evaluate

designs. For instance, in [34] Mur et al. elaborate a mathematical model

that allows the dynamic computation of the optimal number of agents

required to handle a given load in the Highly Organised Team of Agents for

Information Retrieval (HOTAIR) architecture. In [39], Rana et al. propose

a more generic model based on Petri nets, which uses the metrics gathered

Page 39: AgentSpotter: a MAS Profiling System for Agent Factory

4.3. MAS scalability issues 25T

able

4.1:

Com

mon

MA

Ssc

alab

ilit

ypro

ble

ms

and

solu

tion

s

Chara

cteri

stic

Pro

ble

ms

Solu

tions

Nu

mb

erof

agen

tsT

oo

man

yco

ncu

rren

tta

sks

Sch

edu

leag

ents

diff

eren

tly

[13]

Ad

dm

ore

CP

Uif

the

syst

emsc

ales

lin

earl

yS

ched

ule

task

sd

iffer

entl

y(e

.g.

by

pri

orit

y)

Sel

f-tu

nin

gsy

stem

s[3

0,6]

Nu

mb

erof

mes

sage

sM

ore

mes

sage

sth

ann

eces

sary

are

exch

ange

dC

han

geth

eco

ord

inat

ion

pro

toco

lsto

red

uce

the

nu

mb

erof

mes

sage

sM

essa

ges

take

too

mu

chti

me

toar

rive

Ch

ange

the

tran

spor

tati

onm

eth

od

Siz

eof

inp

ut

dat

aor

rule

sd

atab

ase

Ret

riev

alor

pro

cess

ing

tim

eto

oh

igh

Red

uce

the

amou

nt

ofd

ata

Red

istr

ibu

ted

ata

pro

cess

ing

Cac

he

dat

aP

roac

tive

agen

tsR

easo

nin

gta

kes

too

mu

chti

me

orto

om

uch

spac

eC

han

geth

ere

ason

ing

engi

ne

Red

uce

pro

-act

ivit

yR

edis

trib

ute

role

sR

eact

ive

agen

tsT

oo

man

yev

ents

Ch

ange

even

tp

rop

agat

ion

stra

tegi

ese.

g.u

seb

road

cast

[30]

Dec

oup

leag

ent

inte

ract

ion

e.g.

use

ab

lack

-b

oard

Soci

ety

Org

anis

atio

nT

oo

mu

chin

tera

ctio

nA

dop

ta

mor

est

ruct

ure

dor

gan

isat

ion

[29]

Sel

f-or

gan

isin

gag

ents

[6,

51]

Cen

tral

ised

arch

itec

ture

does

not

scal

eu

pR

epli

cate

centr

alco

mp

onen

ts[3

4,13

]T

yp

eof

coor

din

atio

nP

lan

nin

gis

sues

Use

scal

able

pla

nn

ing

algo

rith

ms

[5]

Tas

kd

eleg

atio

nis

sues

Use

the

Con

trac

tN

etp

roto

col

[46,

24]

Ad

min

istr

ativ

eov

erhea

dN

amin

gor

Loca

tion

serv

ice

does

not

scal

eu

pU

seex

isti

ng

mid

dle

war

ese

rvic

es[4

]

Page 40: AgentSpotter: a MAS Profiling System for Agent Factory

26 Performance concepts and solutions in MASs

about a subset of known agents to predict the performance of a much larger

system.

4.3.3 Self-building and self-organising agents

Numerous examples of agent-oriented approaches to scalability issues are

available. Turner and Jennings show in [51] how self-building and self-

organising agents can maintain and adapt their own organisational structure

in order to dynamically improve the global performance of a MAS. A series

of papers about the HOTAIR project [38, 6, 30], a scalable document

indexing system based developed at University College Dublin, provide a

good report on a concrete application of self-organisation to help a system

scale dynamically depending on the load.

4.3.4 Self-tuning agents

The HOTAIR system also contains a good example of self-tuning agents

that optimise their own behaviours depending on system or application

performance metrics [30, 6]. One motivation for self-tuning is to allow an

agent to degrade its individual performance in order to maximise the overall

performance of the system.

4.3.5 Performance analysis of common MAS structures

How does a mesh structure compare to a hierarchical structure? What’s the

impact on the task allocation computational load when an auction protocol

is used instead of a contract net protocol? These questions are examples of

what a MAS designer may ask when selecting an appropriate coordination

structure. A growing number of studies provide answers to this kind of

questions and will ultimately constitute a precious body of knowledge to

guide aspiring MAS designers. Among the most interesting studies, an early

work by Lee et al. [29] demonstrates that hierarchical structures are more

Page 41: AgentSpotter: a MAS Profiling System for Agent Factory

4.4. Facilitating MAS performance case studies 27

scalable than mesh structures; not surprisingly it also shows that an auction

protocol is more computationally expensive than a contract net protocol

because an auction is like a repetitive contract net. In that study, Lee et al.

also conclude that “the benefits of carrying out such investigations become

increasingly important as MASs mature” [29]. We completely agree with

their analysis, but we want to go further and add that carrying out these

performance studies should be greatly facilitated by extending the existing

agent frameworks with the appropriate infrastructure. In other words, as we

explain further in the next section, carrying out MAS performance analysis

should be made much easier in order to encourage the publication of more

studies.

4.4 Facilitating MAS performance case studies

As more MASs are leaving the experimental arena for the production envi-

ronments, performance becomes a greater concern, and it is important to

multiply the investigations into the performance of existing MAS designs.

To represent this type of work, we have selected the following papers:

• In [25], Juhasz and Paul compare the performance of two leading

open source agent platforms, Grasshopper and JADE, running the

same MAS. Their evaluation focuses on the data transfer rate between

agents and the time taken to complete some determined tasks.

• In [12], Cubillos et al. study the performance of a multiagent passenger

transportation planning system implemented with JADE in terms of

planning request average processing time.

In both studies the goal is to provide implementation specific insights into

the performance of a particular MAS so that implementers of similar systems

Page 42: AgentSpotter: a MAS Profiling System for Agent Factory

28 Performance concepts and solutions in MASs

are aware of the potential pitfalls or advantages of the design decisions that

are made and of the platform that is used. Our main interest is not the

content of these studies. From a theoretical point of view, we actually think

that the scope of these studies is narrower than the generic architectural

performance analysis in [29], which we have described earlier (see §4.3.5).

However, from a more practical point of view, these studies are the sign

that MAS platforms and implementations are mature enough to serve as

background for other studies instead of being objects of study themselves.

We believe that the maturity of the MAS platforms and implementations is

an essential prerequisite to allow us to bridge the gap between the theory

and the practice of MAS performance analysis, because it highlights the

need for a generic performance analysis toolkit and it also makes the creation

of such a toolkit technically possible.

In this view, the fact that both systems run under the same platform,

JADE, has motivated our selection of these papers. While the performance

analysis objectives and measurements were completely different, the authors

have probably used the same platform services and API to set up their

experiments and gather the data for their performance analysis. Surely both

teams of researchers had to reinvent the wheel and create themselves the

tools they needed. This is a common problem in the academic world, and

it is not usually viewed as a serious one since it happens in the context of

one-off projects. Yet in production environments where performance analysis

is carried out more routinely as part of the software development life cycle,

MAS engineers will require clear methodologies and general toolkits to avoid

reinventing the wheel, and more importantly, to obtain meaningful results

they can apply immediately to improve their designs.

The availability of such a system might foster more interest in the

Page 43: AgentSpotter: a MAS Profiling System for Agent Factory

4.5. Cougaar 29

evaluation of the performance of existing MAS designs by making the results

easier to obtain. This has happened in the traditional software engineering

community thanks to the introduction of historical profilers like gprof [17],

or performance analysis APIs like ATOM [47] and the Java Virtual Machine

Tool Interface (JVMTI) [49], which have made performance analysis more

accessible for researchers and software engineers.

Without detailing the actual requirements for such a system, we can

nevertheless immediately identify the necessity of a solid performance mea-

surement infrastructure, directly integrated into the agent architecture. A

good example of that architecture is the Cougaar platform that we describe

further in the next section.

4.5 Cougaar, a MAS with a built-in performance

measurement infrastructure

In this section we examine an advanced military grade Java agent platform

called Cougaar, presented earlier in §2.5, which integrates a performance

measurement infrastructure right into the system architecture. While appre-

ciating the great depth and scope of the Cougaar performance measurement

system, we identify the further requirements which we believe are missing

in Cougaar to make it completely relevant in the modern landscape of

multiagent platforms. Indeed, the need for non-expert friendly and time

saving tools has recently risen as a primary concern in multiagent platforms,

which is a direct consequence of the recent development of AOSE.

4.5.1 Requirements

In [19], Helsinger et al. describe the performance measurement infrastructure

they have built into the Cougaar agent architecture. Their approach is based

on a rigorous analysis of the challenges of performance measurement in

Page 44: AgentSpotter: a MAS Profiling System for Agent Factory

30 Performance concepts and solutions in MASs

distributed systems, particularly in distributed multiagent systems, namely

the potentially voluminous amount of performance data, the problem of

coordinated collection of widely distributed data, the necessity of minimising

the impact on the measured system and finally the ability to cope with

changing measurement needs and priorities. To tackle these challenges they

state a clear set of general performance measurement requirements. Their

ideal system should be extensible so as to allow the gathering of performance

data at any place in the system. The impact on system performance should

be minimal. The collected data should be readily available and accurate

for internal or external use. The system should be able to deal with large

amounts of performance data. Eventually the system should be able to

dynamically adapt to changes in measurement needs and priorities.

Helsinger et al. refine these requirements further by classifying perfor-

mance metrics according to their different users (internal or external), usages

(state information, trend analysis, change notification), method of propaga-

tion (in-band or out-of-band) and abstraction level (computer system level,

generic agent-system level, highest level application-specific measurement).

Indeed, the different uses of performance metrics have an impact on how

they are collected and then delivered to their intended recipient.

4.5.2 Design and implementation

The creators of Cougaar claim to have comprehensively incorporated these

requirements into the design and implementation of the Cougaar agent archi-

tecture by developing sub-systems specifically geared towards performance

measurement (“Metrics Service”, “Logging Service” and “Cougaar Events”

mechanism) and allowing performance measurement extension points at

different levels in the system. One particularly interesting feature of Cougaar

agents is their self-tuning ability through an “Adaptivity Engine” under the

Page 45: AgentSpotter: a MAS Profiling System for Agent Factory

4.5. Cougaar 31

supervision of higher level system policies dynamically set by other agents

or by a human operator.

Without detailing the full implementation of the Cougaar performance

measurement infrastructure, we can summarise its main characteristics as

follows:

• Different performance data channels provide metrics of different scopes

and different types. The primary channels are raw polling sensors

at the heart of the system execution engine which gather simple low-

impact data elements such as counters and event sensors that are

triggered when the system steps through predefined places in the code.

The secondary channels provide more elaborate information: status

sensors which summarise the state of Cougaar components; history

(or trend analysis) channel which aggregates and stores performance

data over long running times.

• Computer level metrics (CPU, network, memory usage statistics) are

collected inside the Cougaar execution engine and made accessible as

platform-independent data through the “NodeMetricsService”.

• Agent architecture level metrics are made externally available through

the “BlackboardMetricsService” and can be accessed internally via a

raw sensor channel.

• The message transport service captures statistics on the messages that

flow through it.

• A servlet-based extension mechanism allows the addition of visualisa-

tion plug-ins that feed on the performance metrics data source.

Page 46: AgentSpotter: a MAS Profiling System for Agent Factory

32 Performance concepts and solutions in MASs

• Cougaar provides an event and logging service based on the Log4J

logging framework that can be used to log performance metrics.

• The instrumentation services can be dynamically enabled and disabled

either from the outside or from the inside through a self-steering

mechanism.

• The flexible design of the Metrics service (use of call-backs and listeners)

ensures that it has no impact on the system when it is not in use.

4.5.3 Critique of Cougaar

The comprehensiveness of the Cougaar system approach to MAS performance

measurement has currently no equivalent that we know of. Thanks to a

strong theoretical background, the Cougaar framework seems to cover all

the potential scenarios of performance measurement in a MAS. It is difficult

not to agree with the creators of Cougaar that “the Cougaar performance

measurement system offers a great range of performance data, through a

variety of channels, with dynamically modifiable operating modes, with low

system impact.” [19]

However, if we had to re-implement the Cougaar performance measure-

ment system, while keeping the core infrastructure, we would add a more

explicit provision for an interface that facilitates the creation of performance

analysis tools. In essence, the extended interface should allow tools to gather

run-time metrics data and to control the execution of the MAS at the same

time in a more unified way.

As a concrete example, the interface should allow a tool to connect to a

running MAS, hook up to some events of interest to gather run-time metrics,

then maybe pause and restart the system at some critical points, and finally

stop monitoring the system to present analysis results to the user.

Page 47: AgentSpotter: a MAS Profiling System for Agent Factory

4.6. Discussion 33

4.6 Discussion

4.6.1 Lack of a tool-based MAS performance analysis system

The breadth of our survey has clearly shown the extent of MAS performance

issues and the variety of solutions proposed to solve these issues. We have

also discovered that the tool-based approach to performance analysis we

advocate in this dissertation has been surprisingly neglected in the agent-

oriented paradigm. Yet, as we have demonstrated in Chapter 3, this approach

is favoured in all the other major programming paradigms and is usually

backed up by a solid theoretical foundation. In fact, tools ultimately bring

together performance analysis theory and practice.

The relatively recent focus on the adaptation of mainstream software

engineering practices for the fabrication of agent-oriented systems could

explain the lack of a tool-based approach to performance analysis. Now that

AOSE has become a practical possibility, it is time to integrate the practices

that will help make agent-oriented programming a viable proposition for

mainstream enterprise-level software projects.

To put it another way, we have seen how the MAS community has done

the important work of elaborating a set of concepts to tackle performance

problems in MASs. However, the prescriptions and ideas will only be truly

useful when they can be applied, evaluated or enforced through a set of

practices, methodologies, or even better, tools.

4.6.2 Motivation for a MAS performance analysis toolkit

Modern software performance engineering (SPE) promotes a proactive ap-

proach to performance issues by integrating performance-oriented principles

and best practices into the engineering process from start to end, which

can entail a significant additional upfront cost [45]. In practice, a reactive

Page 48: AgentSpotter: a MAS Profiling System for Agent Factory

34 Performance concepts and solutions in MASs

approach is the norm, where corrective measures are taken only when the

performance issues are discovered late in the process, which usually means

at best a significant redesign, or at worst the failure of the project due to

intractable issues in the upfront design of the system.

As with most software practices, we contend that the reasonable approach

certainly lies half way and should follow the so-called Pareto principle. Our

idea is that for 90% of the code, software engineers should focus primarily

on functionality. For the remaining 10% that are usually linked to non-

functional problems like performance, 90% of the potential performance

issues should be addressed as automatically as possible by the software

itself, either passively by code reuse that integrates the best practices, or

proactively by using agent-oriented software solutions. The remaining 1%

(10% of 10%) can be addressed manually using an easy to use performance

analysis toolkit which gives as much information as possible to software

engineers about the problems at hand.

In other words, a MAS naturally generates complex emergent behaviours

that are likely to exhibit performance issues. Therefore, despite the embed-

ding of the best performance management principles and practices right into

the software, still in a lot of instances the intervention of a human software

engineer is unavoidable. Hence the necessity of providing a comprehensive

toolkit to help software engineers tame the complexity of MAS performance

analysis.

4.7 Towards a set of requirements for a performance

analysis toolkit

In this chapter we have surveyed the MAS performance concepts and solu-

tions and offered an overview of the most common MAS scalability issues in

Page 49: AgentSpotter: a MAS Profiling System for Agent Factory

4.7. Towards a set of requirements for a performance analysis toolkit 35

table 4.1.

Among the cited approaches, we have focused on performance case

studies as a good way to disseminate knowledge about the performance

characteristics of different MAS designs and architectures. After noticing

that this research direction is not as developed as it should be, we have

suggested that the reason for that scarcity could be the current lack of

comprehensive MAS performance analysis system.

Then, as a fundamental requirement for a performance analysis system,

we have identified the necessity of a performance measurement infrastructure

similar to the one built right into the Cougaar agent architecture that we

have reviewed. From our review of the Cougaar system we have concluded

that while a solid performance measurement infrastructure is a good basis for

a performance analysis system, we should also provide an unified interface,

in the form of a toolkit, to facilitate the creation of performance analysis

tools like profilers.

The important point now is to define extensively the requirements of

such a toolkit, which is the main subject of the next chapter.

Page 50: AgentSpotter: a MAS Profiling System for Agent Factory
Page 51: AgentSpotter: a MAS Profiling System for Agent Factory

CHAPTER 5A blueprint for AgentSpotter,

the Agent Factory profiling

system

5.1 Introduction

In the previous chapter we have broadly studied the different aspects of

multiagent systems performance and scalability. We have concluded that

despite the subject’s good theoretical coverage in the literature, and the

existence of performance-minded infrastructures, there is a lack of tools that

facilitate MAS performance issues’ evaluation and analysis from a software

developer’s point of view.

Ideally, agents and MASs should be empowered to deal autonomously

with performance and scalability issues, which is a research area in itself.

However, until generic solutions are found for this problem, the human design

factor remains preponderant, and some specific challenges must be addressed

at the conceptual level and at the infrastructure level to build a toolkit that

37

Page 52: AgentSpotter: a MAS Profiling System for Agent Factory

38 A blueprint for AgentSpotter

will truly help software developers understand and fix performance issues in

MASs.

In this chapter, our goal is to detail our vision of such a toolkit based

on the well-known profiling technique, from the abstract concepts to the

user interface. Our approach is as platform-independent as possible, even

though in the end, we have chosen the Agent Factory platform to implement

a prototype of that system, which we have named AgentSpotter.

We must emphasise that while there are many visualisation programs for

MASs, AgentSpotter is the first desktop profiling system which deals with

agent-oriented concepts and is completely integrated with a mainstream

AOSE development framework.

5.1.1 AgentSpotter core objective

In brief, the core objective of AgentSpotter is to map the traditional concepts

of profiling to agent-oriented concepts in order to build a profiler tool for

MAS developers.

Profiling is indeed the dominant software performance analysis tech-

nique in use in traditional software development paradigms. As we have

explained in Chapter 3, profiling’s key value is the eye-opening quality of the

information it extracts from raw performance data, providing unexpected

insights into what really drives or hampers the performance of a program

and revealing hidden patterns.

5.1.2 Why an agent-specific profiler?

Before describing what the ideal profiler for a MAS should be, we must first

address the main objection that could be expressed against our approach: as

most MASs are based on Java platforms and are built around Java classes,

why cannot we just use the profilers that come with these systems?

Page 53: AgentSpotter: a MAS Profiling System for Agent Factory

5.2. Platform-independent AgentSpotter architecture 39

Our answer is that we need a specific profiling system for multiagent sys-

tems because a Java profiler does not operate at the appropriate conceptual

level. While there can be a technical mapping between objects and agents,

it is usually difficult to connect method calls with specific agent activities

like messaging or deliberation.

Actually, our first attempt at the problem would have been to use the

output of a traditional Java profiler and to try to infer agent-specific activity

from low-level method profiles. The fundamental flaw with that technique is

that it wouldn’t allow us to group the different statistics by agent instance,

so we finally chose to design an agent-specific infrastructure.

5.1.3 Analysis outline

Our analysis starts with a discussion about the required infrastructure for

AgentSpotter. First, we describe AgentSpotter’s abstract, i.e. platform-

independent, architecture. Then we adapt that abstract architecture to the

Agent Factory platform.

Once the overall architecture has been clearly defined, we then set to

detail extensively the requirements for our main visualisation tool called

AgentSpotter Station. Our requirements analysis is particularly focused on

the concepts underlying the novel visualisations that we have introduced.

5.2 Platform-independent AgentSpotter architecture

5.2.1 High-level overview

We provide an overview of AgentSpotter’s abstract architecture in Figure 5.1

using the following graphical conventions:

• Top-level architectural units are enclosed in dashed lines and are titled

in slanted capital letters, e.g. AGENT PLATFORM

Page 54: AgentSpotter: a MAS Profiling System for Agent Factory

40 A blueprint for AgentSpotter

Run-Time EnvironmentProled Application AgentSpotter Service

Proler

System Monitors

AGENT PLATFORM

AGENTSPOTTER STATION (visualisation application)Snapshot File (.aspot)

Session Summary Tables

Space-Time Diagram

Call Graph Tree View

Sessions

Agents

Events

Messages

messages

activity

eventsevents

data

data

queries

QUERY ENGINE

Session summary query

Agent activity query

Messages sent/received query

Call graph query

System activity query

Figure 5.1: AgentSpotter abstract (platform-independent) architecture. Top-level architectural units are enclosed in dashed lines. Self-contained softwarepackages are enclosed in solid lines. Arrows denote data or processinginteractions.

Metrics Service

Agent Management Service

Scheduler

Message Transport Service

Agent Factory Run-Time EnvironmentProled Application AgentSpotter Service

Pro!lerCustom Metrics

Agents

CPU Monitor

Heap Monitor

AGENT FACTORY PLATFORM

AGENTSPOTTER STATION (visualisation application)Snapshot File (.aspot)

Session Summary Tables

Space-Time Diagram

Call Graph Tree View

Sessions

Agents

Events

Messages

data

data

queries

QUERY ENGINE

Session summary query

Agent activity query

Messages sent/received query

Call graph query

System activity query

Figure 5.2: AgentSpotter for Agent Factory concrete architecture: note thatonly the upper part of the diagram is different from Figure 5.1.

Page 55: AgentSpotter: a MAS Profiling System for Agent Factory

5.2. Platform-independent AgentSpotter architecture 41

• Self-contained software packages are enclosed in solid lines e.g Profiler

• Logical software modules (groups of packages) are titled using slanted

capitalised names e.g. AgentSpotter Service

• Arrows denote data or processing interactions e.g. queries-

At the highest level, the AgentSpotter Service should communicate with

the Run-Time Environment to capture the profiling data from a Profiled

Application running inside an Agent Platform. The captured data should

be stored into a Snapshot File which would then be processed by a Query

Engine to generate the input data for AgentSpotter Station, the visualisation

application.

5.2.2 Profiled Application

The Profiled Application should not be directly aware of AgentSpotter

Service’s presence at run-time, just as it should not be aware of a debugger’s

presence for example. Therefore, no direct line of communication should

exist between the AgentSpotter Service and the Profiled Application. This

decoupling ensures that profiling is a concern completely separate from the

normal MAS development process.

In some rare cases it might be necessary for the Profiled Application to

communicate with the AgentSpotter Service. This should be done through

a special infrastructure like the Metrics Service described in §5.3.2.

5.2.3 AgentSpotter Profiler Service

The AgentSpotter Profiler service should monitor performance events gener-

ated by the Agent Platform’s Run-Time Environment. These events should

reflect the different things that happen inside the platform: agent manage-

ment events, agent scheduler activity, messaging, and other platform services

Page 56: AgentSpotter: a MAS Profiling System for Agent Factory

42 A blueprint for AgentSpotter

activity (e.g. agent specific input/output, reasoning engine). As explained

above, the Profiler has no direct line of communication with the Profiled

Application.

Additionally, through modules called System Monitors, the AgentSpotter

Service could optionally record some system performance counters like the

CPU load, the memory usage, or the network throughput, so as to provide

a general context for the Agent Platform performance events.

5.2.4 Sessions and Snapshot File

We call the result of one uninterrupted data capture operation applied

to a Profiled Application a session. A Snapshot File should be able to

record run-time data for multiple sessions. Co-locating sessions this way

will facilitate ulterior manipulation.

Ideally a Snapshot File should be stored as a single file. As a result, a

Snapshot File could be more easily moved around the file system, shared

with other developers, and transported across operating systems.

5.2.5 Query Engine

Basically, snapshot files contain raw instrumentation data. For instance,

every time an agent sends or receives a message, it should be recorded as

a unique “message sent” or “message received” event with a time stamp

and a duration. Similarly, whenever an agent executes a task, it should

be recorded as a single “agent activity” event with a time stamp and a

duration.

Large MAS applications could potentially generate hundreds of per-

formance events per second, which would mean hundreds of thousands of

records over a 10 minute session for example. However, a simple listing of

these events would not give much information about the system. Therefore, a

Page 57: AgentSpotter: a MAS Profiling System for Agent Factory

5.3. Platform-specific AgentSpotter architecture for Agent Factory 43

special data manipulation infrastructure, i.e. a Query Engine, is required to

allow the extraction of meaningful information from the raw instrumentation

data.

The ideal Query Engine should be able to query hundreds of thousands

of performance event records in a few seconds. It should also provide a data

manipulation language similar to SQL (the Structured Query Language

for relational databases) to facilitate the specification of rich and complex

queries.

5.2.6 AgentSpotter Station

The AgentSpotter Station is the profiler front-end application and should

implement the following set of visualisations based on queries provided by

the Query Engine: the Session Summary Tables, the Call Graph Tree View,

and the Space-Time Diagram. We describe AgentSpotter Station in more

detail in §5.4 and onwards.

5.3 Platform-specific AgentSpotter architecture for Agent

Factory

We have applied the AgentSpotter abstract architecture requirements to

Agent Factory in order to produce the concrete, i.e. platform-specific,

architecture illustrated in Figure 5.2.

From an architectural point of view, only the agent platform details

change, as pictured in the upper part of Figure 5.2. In contrast, the

AgentSpotter file processing and visualisation pipeline, as pictured in the

lower part of Figure 5.2, is exactly the same as in Figure 5.1. Essentially,

we are arguing that only the platform-specific data capture apparatus of

AgentSpotter should vary, while the snapshot file processing and visualisation

Page 58: AgentSpotter: a MAS Profiling System for Agent Factory

44 A blueprint for AgentSpotter

modules should stay the same, regardless of the Agent Platform on which

the Profiled Application is running.

The immediate benefit of this architecture is that it will provide a testbed

for comparing the performance of applications running on different platforms,

say JADE and Agent Factory.

5.3.1 Instrumented AF-RTE subsystems

The AgentSpotter instance for Agent Factory requires that the following

Agent Factory Run-Time Environment (AF-RTE) subsystems generate

profiling events: the Agent Management Service, the Scheduler, and the

Message Transport Service. The events generated by these subsystems

should provide enough information to support the standard AgentSpotter

visualisations described in §5.4.

5.3.2 AF-RTE Metrics Service

The standard, “out of the box”, performance data should cover most of

developers’ profiling needs. However, in some special cases, developers

might need to record specific metrics that are not already handled by the

system. Thus a new service, the Metrics Service, should be introduced in

the Agent Factory Run-Time Environment, so that the Profiled Application

can generate custom events that will be observed by the Metrics Service.

The Metrics Service should then forward the events to the AgentSpotter

Profiler in order to get them saved to the Snapshot File.

This seemingly roundabout architecture should enable developers to keep

the instrumentation code in a deployed application with little impact on

the system performance and no direct classpath link with the AgentSpotter

system.

Page 59: AgentSpotter: a MAS Profiling System for Agent Factory

5.4. AgentSpotter Station 45

5.4 AgentSpotter Station

AgentSpotter Station is the profiler front-end application that develop-

ers should use to analyse the result of a MAS profiling session. In line

with the modern profiler tools described in Chapter 3 (see §3.6), it should

be a user-friendly GUI application providing both textual and graphical

interpretations, i.e. visualisations, of the profiling data.

5.4.1 Visualisations

Developers expect profilers to generate flat profiles and hierarchical profiles

(also known as “call graphs”), so AgentSpotter should offer the agent-oriented

versions of these visualisations.

The AgentSpotter version of a flat profile is described in §5.5.

The AgentSpotter version of the hierarchical profile is the Call Graph

Tree View, exposed in §5.6.

Despite their relevance, these basic visualisations fail to fully convey the

distributed complexity of a MAS application; consequently AgentSpotter

should also implement a more advanced visualisation called the Space-Time

Diagram, presented in §5.7.

5.4.2 Side-by-side display of visualisations

AgentSpotter Station should allow developers to open visualisations of

different sessions from different snapshot files at the same time for direct

comparison.

For example, a developer could first profile an application to create a

baseline session. After studying the session, the developer could make a

few changes to the application code or run-time parameters and profile

the application again to generate a second session. The developer would

then open the same visualisation for both sessions, and lay them out in two

Page 60: AgentSpotter: a MAS Profiling System for Agent Factory

46 A blueprint for AgentSpotter

windows side by side on the screen in order to visually compare them, and

immediately see the effects of the changes on the run-time behaviour of the

application.

Along the same lines, developers should be able to open different visuali-

sations of the same session at the same time, i.e. view the Call Graph Tree

View and the Space-Time side by side to see the correlations between them.

5.5 Session Summary Tables

5.5.1 Flat Profile principles

The flat profile is the minimal service that a profiler should provide. For

each profiled application’s identifiable basic execution block, it should clearly

list the following information items:

• the number of times the block was invoked;

• the total amount of time the application spent inside the block;

• the percentage of the total recorded time represented by that amount.

Just by looking at that apparently simple listing ordered by total time

spent, developers should be able to spot bottlenecks immediately i.e. execu-

tion blocks that use up more computing time than they should in absolute

time or compared to other execution units.

5.5.2 Selection of the appropriate execution unit

But what exactly is the appropriate execution unit for a MAS? To answer

that question, we have pitted the OOP execution unit concepts against

their AOP counterparts as shown in table 5.1. Note that we have assumed

agents are implemented as single execution threads, which is the approach

Page 61: AgentSpotter: a MAS Profiling System for Agent Factory

5.5. Session Summary Tables 47

Table 5.1: OOP and AOP execution units mapping

OOP Execution Unit AOP Execution Unit

Process PlatformThread AgentObject N/AMethod N/A

adopted by Java-based Agent Factory and JADE [11, 2]. The table shows

that agents are the lowest-level meaningful execution unit in a MAS.

One could argue that in practice agents are based on objects and carry

out their tasks by invoking methods, but from a conceptual point of view it

is irrelevant.

5.5.3 Vital indicators

In Chapter 4, we have described the vital indicators of a MAS performance

(see §4.2.1 and §4.2.2). AgentSpotter Station should obviously compute and

display these indicators.

5.5.4 Agent statistics

Concretely, AgentSpotter Station should present a table that lists for each

agent the following indicators:

Agent description: name, role, type (deliberative or reactive) of the agent.

Cumulative activity: cumulative computation time used by the agent.

Perception time: perception time used by a deliberative agent.

Action time: task execution time used by a deliberative agent.

Reasoning time: reasoning time used by a deliberative agent.

% session activity: percentage of the global session computation time

used by the agent.

Page 62: AgentSpotter: a MAS Profiling System for Agent Factory

48 A blueprint for AgentSpotter

Number of iterations: number of non-zero duration iterations (i.e. sche-

duling quantums or time slices) used by the agent;

Number of time slice overshoots: number of times where an agent has

overused its time slice allocation;

Maximum and average iteration duration: maximum and average du-

ration of time slices;

Total number of sent/received messages: total number of messages

exchanged by the agent.

5.5.5 Session statistics

Moreover, AgentSpotter Station should provide a session-level statistics

table with the following items:

Total duration: session run-time recorded.

Total activity: amount of computation time recorded over the session.

Total number of messages: number of messages sent or received by the

platform — the numbers could possibly not match in case of inter-

platform communication.

Average number of active agents per second: this number gives an

idea of the level of concurrency of application.

5.5.6 Conclusion

In this section, we have described the requirements for AgentSpotter’s

session-level and agent-level summary tables composing what is called a flat

profile. These statistics should give MAS developers a sense of the order of

magnitude in the profiled application’s run-time parameters. Additionally,

Page 63: AgentSpotter: a MAS Profiling System for Agent Factory

5.6. Call Graph Tree View 49

Figure 5.3: Call Graph Tree View of a fictional C program which removesduplicate lines from a file

developers should be able to spot immediately the most active agents in

terms of computation or messages exchanged.

Although the flat profile is very useful for a preliminary analysis, it must

be used in conjunction with more detailed visualisations like the Call Graph

Tree View or the Space-Time Diagram described in the following sections.

5.6 Call Graph Tree View

5.6.1 Overcoming the limitations of Summary Tables

Summary tables give developers a sense of how much time agents have spent

doing things, but they do not give much detail about the chain of events

that has led the agents to do these things.

Page 64: AgentSpotter: a MAS Profiling System for Agent Factory

50 A blueprint for AgentSpotter

Suppose for example that an agent is found to use more that half of

a MAS’s computation time while it was expected to use only a quarter.

Looking at the agent’s internal resource use could give a good idea of the

cause of the problem e.g. “I think the agent X took too much time doing a

bubble sort on a list of 10000 integers”.

However, we have shown in Chapter 4 that analysing the behaviour of

an agent in isolation is not sufficient to understand fully the cause of a MAS

performance problem. Indeed, the real complexity of a MAS comes from

the distributed interactions between the agents.

Consequently, AgentSpotter should implement a visualisation that allows

developers to understand the impact of interaction on the performance of a

MAS. In traditional profilers, this kind of visualisation is called a hierarchical

profiler or call graph (see Chapter 3, §3.5.1).

5.6.2 Traditional call graph

A call graph is basically a tree displaying all the calls made by a program.

These calls are grouped by call target, and are annotated with the cumulative

time spent in each call target.

The example in Figure 5.3 shows a typical call graph of a completely

fictional C language program which is supposed to remove duplicate lines

from a file. The root node is the main function and it represents 100% of

total execution time. The idea is that each tree’s level displays recursively all

the calls made by the current node as child nodes along with the percentage

of time they represent and so on until the top of the call stack is reached.

Therefore, if main is calling process file, it is represented as a sub-tree with

leaves describing its own calls to bubble sort, load file and remove duplicates.

The call graph tree view’s key benefit is the extended context it gives to

the performance information. For instance, our simple example reveals that

Page 65: AgentSpotter: a MAS Profiling System for Agent Factory

5.6. Call Graph Tree View 51

Agent A

Agent B

Agent X

message Mα at time α

time

message MΩ at time Ω

impact of Mα

Legend:

Message event

Agent activity

Agent life line

impact of MΩ

α Ωβ

Figure 5.4: Agent message impact concept diagram

the program spends 90% of its time processing a file. The tree shows that

one of the top-level function process file’s callees, the bubble sort operation,

eats up 80% of its caller time. A flat profile would have shown the time

for these functions separately without explicitly showing the hierarchical

link between them. In other words, what matters is to know the “where” to

understand why the “how long” is a problem.

5.6.3 Agent-oriented call graph model

At this point, the problem of mapping a traditional programming concept

to an agent concept arises again. Specifically, what is the equivalent of a

function call for an agent? This is a much more difficult question than it

seems, and would necessitate a long discussion that we have decided not to

develop in this dissertation for space reasons. Instead, we invite the reader

Page 66: AgentSpotter: a MAS Profiling System for Agent Factory

52 A blueprint for AgentSpotter

to allow us to make a conceptual leap without a proper demonstration.

In short, we assert that we can measure TMα,B, the impact of a message

Mα sent from an agent A to an agent B and received at time stamp α,

by measuring the total amount of computation time used by the agent B

until agent B receives a message MΩ from another agent X at time stamp

α ≤ Ω. Let b be the duration of an activity by agent B at time stamp t

where α ≤ t ≤ Ω. The impact of message Mα on agent B, TMα,B, is then

given by the recurrent equation:

TMα,B =Ω∑t=α

bt (5.1)

In Figure 5.4 we have tried to summarise this concept in a graphical form.

The diagram clearly shows that the three activity stars that lie between α

and Ω make up the total impact of Mα on agent B. Note that the outgoing

message at time stamp β does not break the computation sequence.

It is now easy to determine the total impact Tx,y of all the messages

sent by a given agent x to another agent y. Let M be the total number

of messages sent, 1 ≤ m ≤ M a single message impact identifier, αm the

reception time stamp of message m from x to y, and Ωm, where αm ≤ Ωm,

the next reception time stamp message coming right after m from any other

source. The total impact Tx,y is then given by the equation:

Tx,y =M∑m=1

Ωm∑t=αm

bt (5.2)

By applying the equations recursively, we can compute the total impact

Tx of an agent x on N other agents numbered 1 ≤ a ≤ N as follows:

Tx =N∑a=1

Ma∑m=1

Ωm∑t=αm

bt (5.3)

Page 67: AgentSpotter: a MAS Profiling System for Agent Factory

5.6. Call Graph Tree View 53

Figure 5.5: AgentSpotter Call Graph Tree View levels

Finally, the total impact TS of all the K agents numbered 1 ≤ k ≤ K of

a session S is given by the equation:

TS =K∑k=1

Nk∑a=1

Ma∑m=1

Ωm∑t=αm

bt (5.4)

It must be noted that the total activity time AS of the session S is given

by the equation:

AS = TS +K∑k=1

αk0−1∑t=αS

bt (5.5)

where αS is the first recorded time stamp in session S and αk0 the time

stamp of the very first message received by agent k. To put it differently,

the total impact for each agent can be computed only after it has received

its first message.

A traditional call graph’s depth can be as deep as the deepest function

call stack. In contrast, the graph representation of our model is a fixed-depth

tree.

5.6.4 Call graph model application

The model we have just defined is the basis for the graphical representation

that AgentSpotter Station should provide to help developers understand

better the impact of agents interactions on a MAS performance.

Page 68: AgentSpotter: a MAS Profiling System for Agent Factory

54 A blueprint for AgentSpotter

This conceptual model deals with the session level, the emitter agent level,

the receiver agent level and the message level. The graphical translation

of the model, outlined in Figure 5.5, should be a tree view representing

the levels we have previously enumerated plus an additional level for the

FIPA ACL message content. A message content is defined as a performative

plus an expression e.g. “request:doSomeThing(123)”. This additional level

should give developers necessary contextual information for the messages.

Of course the model must be slightly extended to accommodate for this new

level, but this is left as an exercise for the reader.

The session at the root of the tree should add up to 100% of all emitter

agents’ impact as defined by equation 5.4. Then at each level, each node

should recursively total the impact of its child nodes down to the message

leaf nodes. These leaf nodes simply report their impact as defined by

equation 5.1. More precisely, at each level, for each node, the following

values should be displayed:

Label: informative text associated with the node. The structure of the

label depends on the level as follows:

• session: “capture date and time - duration”;

• emitter agent: “from: agent id”;

• sender agent: “to: agent id”;

• FIPA ACL: “performative: contents”;

• message: “sent: time stamp rec: time stamp”.

Total impact time: sum of impact times of all the current node’s children.

% parent time: percentage of the current node total impact time over the

node’s parent total impact time.

Page 69: AgentSpotter: a MAS Profiling System for Agent Factory

5.6. Call Graph Tree View 55

% session time: percentage of the current node total impact time over

the session total impact time.

Ideally, developers should be able to order the intermediary tree levels

differently so as to produce different call graph interpretations. For example,

moving the FIPA level right above the emitter agent level would list for

each FIPA ACL entry their total impact for all the emitter/receiver pairs.

5.6.5 User interface requirements

Even if it has a fixed depth, a call graph tree view could potentially be

very wide at the leaf level for sessions that produce thousands of messages.

Therefore, to help developers navigate easily through the tree, AgentSpotter

Station should offer an advanced tree navigation user interface that would

develop only the part of the tree that is currently explored to reduce the

visual clutter. The currently explored part of the tree should be highlighted

in a different colour to give the developer a visual feedback.

Moreover, to speed up information retrieval, a special search feature

should allow developers to enter a keyword (an agent name or a performative

for example). Doing so would have the effect of highlighting in a special colour

all the visible nodes which contain that keyword, improving significantly the

visual retrieval speed of a node.

Ideally, to enable developers to locate items even more quickly, zooming

and panning around the tree view should be possible.

5.6.6 Conclusion

To summarise, in this section we have proposed a new visualisation, the

call graph tree view, in order to overcome the main limitation of the Agent-

Spotter flat profile, namely the lack of detailed information about the agents

interactions.

Page 70: AgentSpotter: a MAS Profiling System for Agent Factory

56 A blueprint for AgentSpotter

Figure 5.6: Lamport Space-Time Diagram extracted from [28] ( c©1978ACM).

After exposing the concept of call graph in a traditional programming

context, we have then mapped it into an agent-oriented concept based on

the idea that when an agent A sends a message to an agent B, its impact

TMα,B can be measured by the equation 5.1.

We have then extended this notion to a tree model with multiple levels:

session, message emitter, message receiver, message.

Finally, we have described the advanced user interface that would allow

developers to interact with this model in the form of a zoomable and

searchable tree view.

5.7 Space-Time Diagram concept

5.7.1 Motivation

The Call Graph Tree View is an information-rich visualisation that should

make AgentSpotter easily comparable to existing visual Java profilers. Fur-

thermore, it is fully agent-oriented and should provide real insights into the

Page 71: AgentSpotter: a MAS Profiling System for Agent Factory

5.7. Space-Time Diagram concept 57

performance behaviour of a MAS.

Nonetheless, we believe the call graph does not fully capture the richness

of the concurrency and the distribution of performance in MAS systems.

To put it bluntly, the call graph reveals the link between the “where” (the

agents), the “what” (the messages) and the “how long” (the performance

timings), but fails to convey a sense of the “when” in an obvious way.

Consequently, we need to add to AgentSpotter a comprehensive graphical

representation that conveniently conjugates space and time, if we want to

picture out how phenomenons like message passing and performance timings

– the “what”, evolve across time – the “when”, among a community of

distributed agents – the “where”.

5.7.2 Lamport Diagram

Distributed systems specialists commonly use a graphical representation

called a “space-time diagram” or “Lamport Diagram”. It was made popular

by L. Lamport in the seminal paper “Times, Clocks, and the Ordering of

Events in Distributed Systems” [28]. In his paper, L. Lamport suggests the

space-time diagram as a useful view for the “happened before” relation that

he defines as follows:

Definition. The relation “→” on the set of events of a system

is the smallest relation satisfying the following three conditions:

(1) If a and b are events in the same process, and a comes before

b, then a→ b. (2) If a is the sending of a message by one process

and b is the receipt of the same message by another process, then

a→ b. (3) If a→ b and b→ c then a→ c. Two distinct events

a and b are said to be concurrent if a→ b and b 6→ a. We assume

that a → a for any event a. (Systems in which an event can

Page 72: AgentSpotter: a MAS Profiling System for Agent Factory

58 A blueprint for AgentSpotter

happen before itself do not seem to be physically meaningful.)

This implies that → is an irreflexive partial ordering on the set

of all events in the system.

The author then extends the partial ordering defined by the “→” relation

to a total ordering which he uses to describe a distributed algorithm to

synchronise a system of clocks.

To simplify a little bit, synchronising clocks this way allows the definition

of a “global” reference clock which ticks at the same time for all the processes

of the system. Eventually, such a global clock makes the representation of

distributed processes in a common space-time diagram possible, since a given

clock tick means the same time for all the processes. A typical Lamport

diagram extracted from [28] is pictured in Figure 5.6. Vertical lines represent

process time lines, while the dashed horizontal lines mark the ticks of the

global clock. The squiggly arrows denote message passing events. Thanks

to the synchronised global clock ticks’ markings, the causality relations

between the different distributed concurrent events appear very clearly.

5.7.3 Principles

The Lamport Diagram should constitute the core for the comprehensive

MAS performance visualisation we need. In contrast with the flat profile

and the call graph, our goal is not to reduce the data to a few meaningful

dimensions.

On the contrary, our plan is to display as much detail and context as

possible about a MAS performance, and let developers pan the view around,

zoom in and out in order to reveal hidden patterns or focus on minute

details.

As an analogy, it is like having a zoomable satellite view of the city of

Page 73: AgentSpotter: a MAS Profiling System for Agent Factory

5.8. Space-Time Diagram user interface 59

Metrics Service

Agent Management Service

Scheduler

Message Transport Service

Agent Factory Run-Time EnvironmentPro!led Application AgentSpotter Service

Pro!lerCustom Metrics

Agents

CPU Monitor

Heap Monitor

AGENT FACTORY PLATFORM

Session Time Line

CPU Line

Agent Life Line

intra-platform message lines

Agent Life Line

information window on mouse hover

External Platform Life Line

Heap Monitor

Message Transport ServiceMessage Transport ServiceMessage Transport Service

inter-platform messages linestimed events (with duration)

“Bird’s Eye View”

♣♦♠

simple events (no duration)

Session Time Line

CPU Line

Agent Life Line

Agent Life Line

External Platform Life Line

♣♦♠

curre

nt v

iew

po

rt

Figure 5.7: AgentSpotter Space-Time Diagram specification annotated withAgentSpotter for Agent Factory infrastructure links (see 5.2).

Dublin with a map view overlay showing the street names, the points of

interest and some geolocated statistics (e.g. number of pubs per square

kilometre): the mix of raw visual data and value added geolocated data

generates much more information.

Page 74: AgentSpotter: a MAS Profiling System for Agent Factory

60 A blueprint for AgentSpotter

5.8 Space-Time Diagram user interface

Basically, the visualisation should appear as a Lamport diagram where

the time flows horizontally towards the right instead of flowing vertically

towards the top. At the risk of stating the obvious, the distributed processes

pictured in the diagram should be agents. Moreover, developers should be

able to interact with the diagram using the mouse.

It can be awkward to try to describe a complex visual construction using

words only, so we invite the reader to use Figure 5.7 as a visual map to our

AgentSpotter Space-Time Diagram specification. In the following sections,

we describe the required diagram elements, starting from the top of the

picture.

5.8.1 Session Time Line

Represents the cadence of the global system clock.

• A pixel point on the time line should map to a millisecond precision

time stamp. The pixel coordinate of a given time stamp is obtained

by applying a certain factor to avoid stretching the display too much

(1 pixel for 25 milliseconds seems to produce a good display).

• The session time line should always be visible whatever the position and

the scale of current view port, so as to provide an essential temporal

context.

• Helpful labelled tick marks should be drawn so that developers can

quickly make an approximation of the current time stamp at a given

position on the time line. Long ticks should mark the seconds, and the

current time should be displayed every 5 seconds. Shorter tick marks

should be displayed every 250 ms.

Page 75: AgentSpotter: a MAS Profiling System for Agent Factory

5.8. Space-Time Diagram user interface 61

• On mouse hover, a pop up information window should indicate the

exact time stamp at the point under the mouse down to the millisecond.

5.8.2 CPU Line

Optional graphical plot of the global CPU load of the host system during a

session.

• A vertical gradient going from green (low CPU usage) to red (high

CPU usage) should give a sense of the system load.

• On mouse hover, a pop up information window should indicate the

exact CPU load percentage at the point under the mouse.

5.8.3 Agent Life Lines

Represent all the performance events and message passing activity that

occurred for a given agent during a session.

• The life line should start only at the time stamp where it was created

or resumed in order to make fluctuations in a MAS population more

apparent.

• The life line area grey scale colour percentage should be proportional

to the agent’s contribution to the global system activity: the more

an agent uses computation time, the darker its life line area’s colour

should be.

• By default, agent life lines should appear from top to bottom in

descending order of total computation time. Ideally, developers should

be able to change this default order by moving the life lines up or

down in order to visually bring together life lines with particularly

interesting interactions.

Page 76: AgentSpotter: a MAS Profiling System for Agent Factory

62 A blueprint for AgentSpotter

Time

% t

ime

sli

ce

50%

75%

100%

0%

1 time slice

150% of time slice (exceeded allocation: RED)

80% of time slice (over 75%: ORANGE)

50% of time slice (under 75%: GREEN)

proportional dimensions

Figure 5.8: Agent Factory agent activity representation in the Space-TimeDiagram

• The life line caption (the agent name) should always be visible whenever

a part of the life line is visible so as to provide an essential contextual

information.

• When the mouse pointer hovers on the life line caption, a pop up

information window should display the agent statistics appearing in

the flat profile (see §5.5.4).

• Developers should also be able to hide temporarily certain life lines in

order to reduce the visual clutter.

5.8.4 Performance Events

All the performance events related to an agent should be displayed along its

life line as they happen.

• Events that have a time stamp only, i.e. simple events, should appear

Page 77: AgentSpotter: a MAS Profiling System for Agent Factory

5.8. Space-Time Diagram user interface 63

as little glyphs centred at the location corresponding to their time

stamp of occurrence. For example, “message received” events could

be displayed with a little envelope glyph.

• Events that have a duration, i.e. timed events, should be drawn with a

size proportional to their duration and should provide as much visual

information as possible.

Consider an Agent Factory agent activity event: it has a measured

actual execution time and an allocated “time slice” duration that the

agent is expected not to exceed. If an agent exceeds its time slice

allocation, it may prevent other agents from getting some CPU time.

Thus, it would be interesting to represent visually the fact that an

agent activity has exceeded its time slice allocation in a way that

would underline its punishing effect on performance. Our idea is to

make AgentSpotter use colour coded rectangles to represent agent

activity.

As illustrated in Figure 5.8, the rectangle size should be proportional

to the event duration and its allocated time slice use percentage.

The colour code should indicate how the agent has used the time

slice. Green would mean that it has used only 3/4 (75%) of its time

allocation. From 75% up to 100%, orange would signify that the agent

has nearly used up most of its allocated time. When the agent exceeds

its time allocation, the rectangle should be painted red. Figure 5.8

demonstrates how different time slice usages should be represented:

50% (green), 80% (orange), and 150% (red).

• On mouse hover, a pop up information window should display specific

details about the event.

Page 78: AgentSpotter: a MAS Profiling System for Agent Factory

64 A blueprint for AgentSpotter

• Other representations should be invented for other performance events,

which could be the subject of an ulterior study. Imagine for example

an extended activity event display for an Agent Factory BDI agent

that would show the proportion of reasoning, perception and action

for each event.

5.8.5 Message Lines

All communication between agents should be rendered as message lines that

link the emitter with the receiver.

• A little arrow end should be added to the message line to show clearly

the direction of the message.

• Messages that stay inside a platform (intra-platform) should be distin-

guished from messages that are going to or coming from outside the

platform (inter-platform).

• On mouse hover, a pop up information window should display all the

FIPA message headers and its content.

5.8.6 External Platform Life Line

When agents communicate with an external platform, a placeholder life line

for the platform should be drawn.

• The AgentSpotter infrastructure is mono-platform only, which means

that it cannot profile two different platforms at the same time. Yet

it should be possible to profile multiple communicating platforms at

the same time by running profiling sessions separately. The resulting

data could then be merged a posteriori through the Query Engine.

The main task of the merge operation would be to synchronise the

Page 79: AgentSpotter: a MAS Profiling System for Agent Factory

5.8. Space-Time Diagram user interface 65

timestamps in the different sessions using a Lamport clock as explained

earlier (see §5.7.2). For space reasons we won’t examine this process

in detail, although it is well worth further studying.

• As the platform life line is a placeholder only, messages lines that

are linking to it are assumed to have the same sending and receiving

timestamps. As a result, they should always render as straight lines.

5.8.7 Tooltip Window

Most diagram elements should pop up a yellow information window (also

called Tooltip Window) to display relevant data about the currently pointed

location. This seemingly minor feature is in fact the most efficient way to

provide developers with the right information in the right context.

5.8.8 Bird’s Eye View

Agent space-time diagrams for long running sessions could potentially span

tens of thousands of pixels. For example it would require (10∗60∗1000)/25 =

24000 pixels across to display a 10-minute long session, which is equivalent

to more than 24 screenfuls of a 1024 pixels wide monitor.

Thus, to facilitate the navigation around such a large view, a zooming

and panning facility is an imperative requirement.

Moreover, AgentSpotter should implement a reduced global view in low

resolution of the full diagram, called a “bird’s eye view”, where developers

can quickly select areas to explore by moving around a little rectangle

representing the current view port. The view port is depicted as a transparent

blue area on Figure 5.7: its current position is perfectly mirrored by the

little blue rectangle in the bird’s eye view window.

Page 80: AgentSpotter: a MAS Profiling System for Agent Factory

66 A blueprint for AgentSpotter

5.8.9 Links with the AgentSpotter for Agent Factory

infrastructure

At the risk of overloading the reader with too much information, we have

traced on Figure 5.7 the links between the AgentSpotter data capture

infrastructure for Agent Factory and the space-time diagram. The dashed

arrows show how the data coming from each Agent Factory Run-time

Environment sub-system, as well as the data coming from system monitors

(CPU usage), is represented on the diagram. Our point is to demonstrate

the incredible richness of our visualisation.

Experimented human computer interface (HCI) designers will surely

object that such density could lead to a confusing picture. We will address

this objection later in our evaluation of the system (see Chapter 7).

5.8.10 Conclusion

In this section, we have developed the requirements for AgentSpotter’s piece

de resistance: the Space-Time Diagram. It is inspired from the Lamport

Diagram which we have extended to agents, and made interactive with the

aim of providing a rich contextual visualisation for a MAS profiling session.

The diagram’s key value is the sheer amount of information it manages to

convey. On a single diagram, developers should be able to visually connect

together the performance and interaction history of a MAS, both in space

and in time. They should also be able to instantly zoom from a global

overview of the system down to a millisecond detail.

We strongly believe that the diagram’s rich information presentation

and advanced interactivity features will be essential tools to inform MAS

designers better about performance and the behaviour of their system.

Page 81: AgentSpotter: a MAS Profiling System for Agent Factory

5.9. Summary 67

5.9 Summary

In this chapter we have extensively described the requirements for Agent-

Spotter, our comprehensive MAS profiler, in a way that can be used as a

blueprint to develop a software prototype.

First, we have managed to define an abstract architecture that should

allow AgentSpotter’s development for another Java-based MAS framework

like JADE.

Second, we have introduced a low-impact, decoupled profiling infrastruc-

ture specifically tailored for Agent Factory. Our infrastructure should make

it possible for developers to instrument without difficulty their systems and

store data capture sessions into readily manipulable snapshot files.

More significantly, we have explained the concepts underlying the Agent-

Spotter Station application, and detailed the visualisation tool suite it should

offer.

Initially, we have presented the most basic visualisation tool, called Flat

Profile or Summary Tables. It should be used to get a preliminary report

on agents vital performance indicators.

Then, for more detailed performance analysis, we have imagined more

advanced tools: the Call Graph Tree View and Space-Time Diagram.

The Call Graph Tree View is a multi-level tree representation of the

impact of agents interactions on other agents performance. Its main use is

to point out interesting interaction-related performance issues.

Finally, the Space-Time Diagram, based on the well-known Lamport

Diagram, is the richest visualisation of the suite. In essence, it displays a

MAS’s full history of performance and message passing events over a given

profiling session. The diagram fully exploits the human mind’s ability to

process efficiently dense information when it is presented in a spatio-temporal

Page 82: AgentSpotter: a MAS Profiling System for Agent Factory

68 A blueprint for AgentSpotter

context. It should allow MAS developers to obtain unexpected insights into

the performance behaviour of their systems.

In short, we have specified a novel system that will truly help MAS

developers understand better the performance of their programs.

In the next chapter we will show how we have used our requirement

analysis as a blueprint for developing a prototype of the AgentSpotter

profiling system for the Agent Factory platform.

Page 83: AgentSpotter: a MAS Profiling System for Agent Factory

CHAPTER 6AgentSpotter design and

implementation

6.1 Introduction

Following the requirements’ analysis elaborated in the previous chapter,

we have implemented a fully working beta version of AgentSpotter for the

Agent Factory platform. Our goal in this chapter is to expose the high

level design and implementation details of AgentSpotter, then present and

comment on screen shots of the final application. In fact, we have managed

to implement nearly all the features we envisioned in Chapter 5, so we will

broadly follow the same structure to describe the resulting system.

The first part of this chapter is devoted to the profiler infrastructure. We

explain how we have extended the Agent Factory Run-Time Environment

to support our profiler service. This is followed by a discussion about

the profiler service, and the implementation of the profiling data capture

pipeline.

The second part focuses on AgentSpotter Station, the profiler visualisa-

69

Page 84: AgentSpotter: a MAS Profiling System for Agent Factory

70 AgentSpotter design and implementation

tion tool suite, and consists mostly in commenting on screen shots of the

application.

6.2 AgentSpotter implementation architecture: an overview

Building upon the concrete architecture we defined in Chapter 5 (see §5.3), we

have implemented the software system illustrated in Figure 6.1. Naturally,

we have based this implementation diagram on Figure 5.2. Besides the

graphical conventions explained in §5.2.1, this new diagram adopts the

following additional ones:

• Software modules are decorated with their corresponding Java package

or class names, e.g.

com.agentfactory

Agent Factory Run-Time Environment

• Open source off-the-shelf components that we have used are identified

by their logo e.g.

• Software modules that we have developed from scratch are titled in

red e.g. AgentSpotter Service

To implement the core AgentSpotter profiling infrastructure, we have

slightly extended the existing Agent Factory Run-time Environment to

generate performance-related events at some strategic points in the Agent

Management Service, the Scheduler, and the Message Transport Service.

These extensions use classes introduced by the new Metrics Service. The

run-time subsystems communicate profiling events to the Profiler.

The AgentSpotter Service relies on the AgentSpotter File Manager to

store the profiling data in a Snapshot File. AgentSpotter snapshot files

are actually single-file databases managed by the public domain SQLite

Database Engine [22]. As a result, profiling data is stored as queryable

relational database tables.

Page 85: AgentSpotter: a MAS Profiling System for Agent Factory

6.2. AgentSpotter implementation architecture: an overview 71

SQLite Database Engine

ANTLR StringTemplate

Agent Management Service

Scheduler

Message Transport Service

Agent Factory Run-Time EnvironmentProled Application AgentSpotter Service

AgentSpotter File Manager

TraceEvent ClassPro!ler

AgentsCPU Monitor

AGENT FACTORY PLATFORM

AGENTSPOTTER STATION RCP APPLICATION

Snapshot File (.aspot)

Session Summary Tables

Space-Time Diagram

Call Graph Tree View

Se

ssio

n S

um

ma

ry T

ab

les

Sp

ace

-Tim

e D

iag

ram

Ca

ll G

rap

h T

ree

Vie

w

Sessions

Agents

Actually a SQLite database le

Events

Messages

data

SQL Templates SQL

data

TraceEvent’s

queriesQUERY ENGINE

Session summary query

Agent activity query

Messages sent/received query

Call graph query

System activity query

Con!guration !le APP.CFG

...PLATFORM_GUI com.agentfactory.

agentspotter.AgentSpotter

...

Netbeans Platforms APIs

Piccolo2D

com.agentfactory.agentspotter

com.agentfactory.service.metricscom.agentfactory.agentspotter.AgentSpotter

com.agentfactory.agentspotter.cpuinfo

com.agentfactory

com.agentfactory.agentspotter.file

AgentSpotter Visualisation Components

com.agentfactory.agentspotter.visualisation

org.openide

com.agentfactory.agentspotter.module

org.prefuse

org.piccolo2d

org.sqlite

org.antlr.stringtemplate

data

DDL

Figure 6.1: AgentSpotter implementation architecture

Snapshot files are processed by AgentSpotter Station visualisation tool

suite, which is implemented as a “rich client” application based on the

Netbeans Platform APIs [33]. In an IDE-like environment, AgentSpotter

Station encapsulates the AgentSpotter Visualisation Components. These

components use the AgentSpotter File Manager to extract the profiling in-

formation from the snapshot files, and exploit the advanced 2D visualisation

features of the Piccolo2D [1] and the prefuse [18] graphical toolkits.

Page 86: AgentSpotter: a MAS Profiling System for Agent Factory

72 AgentSpotter design and implementation

6.3 AgentSpotter profiling infrastructure

6.3.1 AF-RTE performance data channels

Our first implementation task for the AgentSpotter profiler was to ex-

tend the Agent Factory Run-Time Environment (AF-RTE). Thanks to the

Agent Factory framework’s modular architecture, it was easy to identify

the strategic points in the code where the instrumentation code had to be

injected (the “point-cuts” in aspect-oriented programming terminology).

Concretely, we have introduced a new com.agentfactory.agentspotter.

service.metrics package which simply contains a new data TraceEvent

class. Thus, the AF-RTE sub-systems were made to generate the profiling

events by creating TraceEvent instances.

As was done in Cougaar [19], we have applied the Publisher-Subscriber

pattern in order to implement the data communication channels between

the AF-RTE and the Profiler service. In effect, the AF-RTE sub-systems

can publish performance data regardless of the actual presence of a profiler.

Then, when it is activated, the Profiler service can subscribe to the different

well-known performance data channels in order to collect trace events and

persist them to an AgentSpotter snapshot file.

The protocol we have used is very straightforward and is based on simple

hierarchical slash-separated (“/”) string event identifiers. For example, an

agent scheduler activity event is published as a TraceEvent instance with a

Type property value of “scheduler/activity” where the first path element,

“scheduler”, denotes the origin of the event, and the rest, “activity”, indicates

the nature of the event. Table 6.1 presents the full list of the events currently

published by the AF-RTE. We expect this list to grow as more data channels

will be added to the system.

Note that for lack of time, we did not implement the Metrics Service

Page 87: AgentSpotter: a MAS Profiling System for Agent Factory

6.3. AgentSpotter profiling infrastructure 73

Table 6.1: TraceEvent types generated by the AF-RTE sub-systems

AF-RTE Sub-System Event Type

Agent Scheduler scheduler/activityMessage Transport Service message/send

message/receiveAgent Management Service agentContainer/resume

agentContainer/suspendagentContainer/terminate

Metrics Service custom/simpleEvent(to be implemented) custom/timedEvent

which is supposed to handle custom profiling data.

6.3.2 FIPA ACL message identification

One interesting problem we had to solve is how to link up a send event

of an ACL message with the corresponding receive event. The obvious

solution is to identify uniquely a message and use the same identifier for

the send and receive events. Contrary to our expectations, however, the

FIPA ACL Message Structure Specification [15] says nothing about a unique

message identification field. As a stopgap solution, we have decided to

use the conversation identifier field to store the required unique message

identifier.

The drawback of this solution is that some applications might need to

use the conversation identifier field for their own purpose. Fortunately, from

a technical point of view, this new problem can be solved easily by having

the Message Transport Service (MTS) use a special wrapper mechanism

which would allow the addition of system information to a message for

instrumentation purposes 1. But, from a conceptual point of view, it is not

1Very simply, at emission time, the MTS could prepend the conversation identifierfield with a message unique ID followed by a special separator. At reception time, theMTS would remove the message ID prepended to the field, and hand over the originalmessage to the receiving agent. This solution should be readily adaptable to differentagent platforms.

Page 88: AgentSpotter: a MAS Profiling System for Agent Factory

74 AgentSpotter design and implementation

really a clean solution. Ideally, the FIPA ACL message standard should

include a “message identifier” field reserved for system use.

6.3.3 AgentSpotter profiling service

The Profiler service contains two parts: the profiler class, AgentSpotter,

which is implemented as an Agent Factory platform service 2, and an

operating system specific CPU usage monitor.

Profiler

As explained in the previous section, the AgentSpotter class is a perfor-

mance data channel subscriber. Being an Agent Factory platform service,

it can be directly plugged into an existing Agent Factory application by

following these simple steps:

1. Add the required AgentSpotter JAR files to the application classpath.

2. Add a single line to the platform configuration file:

PLATFORM GUI com.agentfactory.agentspotter.AgentSpotter

3. Run the application normally to generate the profiling data.

When AgentSpotter Service is started up, it opens a data connection using

the File Manager, subscribes to the AF-RTE performance data channels,

and then waits for the data publishers to prompt it about a channel update.

When prompted, it pulls down the updated data and forwards it with

no further processing to the File Manager. The loop goes on until the

2According to FIPA standards, the purpose of a platform service is to “providesupport services for agents” [16]. Platform services are at the core of the Agent Factoryinfrastructure: they are the recommended mechanism for implementing new systemfunctionality [9].

Page 89: AgentSpotter: a MAS Profiling System for Agent Factory

6.4. AgentSpotter File Manager 75

application terminates. The most important point is to keep processing time

to a minimum, so as to avoid affecting the system performance too much3.

CPU Monitor

To demonstrate a system monitor implementation, we have developed a CPU

monitor that records the CPU usage percentage every second. Accessing

system information like the current CPU load in Java requires some operating

system specific code. Therefore, the CPU monitor is an optional feature. At

the moment, it works only for the Windows and Linux operating systems.

Nonetheless, the additional contextual information provided by the CPU

monitor is worth the effort. That is why we plan to add support for more

operating systems like Mac OS X in the future.

6.3.4 Running the Debugger and the Profiler together

Interestingly, because AgentSpotter is a mere Agent Factory platform service,

it is entirely possible to run it along with the Agent Factory Debugger tool [7].

This combination can make a lot of sense in some scenarios where developers

want to keep a trace of a debugging session.

6.4 AgentSpotter File Manager

The File Manager is the bridge between the data capture infrastructure and

the visualisation toolkit. In accordance with our requirements, we needed

a compact and reliable snapshot file format, and a queryable and flexible

data retrieval system. Given the fact that profiling data is highly structured,

and that we do not expect shared access to a snapshot file, it was obvious

that a single file database system was the best basis for our File Manager.

3For lack of time, we could not conduct an evaluation of the actual impact of theprofiler at run-time. In practice, the instrumentation does not seem to significantly slowdown the profiled applications.

Page 90: AgentSpotter: a MAS Profiling System for Agent Factory

76 AgentSpotter design and implementation

Informed by our own experience with similar systems, we have selected

the SQLite Database Engine [22]. SQLite is uncomplicated, very fast, and

implements a sizable subset of the SQL-92 standard.

Our approach is to be contrasted with a log file based persistence solution

like the one used by Cougaar [19]. Whilst a log file seems easy and quick to

produce, we expect subsequent automated information retrieval operations

to require some significant programming efforts.

It must be stressed that the File Manager does not directly expose

to its software clients its acquaintance with a particular database system.

Instead, it provides an API that deals with domain specific data record

classes representing profiler data like SessionRecord, AgentRecord, and

MessageRecord.

We won’t examine the characteristics of the File Manager in further

detail, since we have already explored the rationale of a solid file and query

management sub-system (see §5.2.4 and §5.2.5).

In conclusion, AgentSpotter records profiling data in snapshot files that

are in reality SQLite database files4. These compact database files are easy

to query, manipulate and share with other developers or systems.

6.5 AgentSpotter Station

From the developers’ point of view, the AgentSpotter Station desktop

application is the profiler central attraction. It can be seen in the screen

shot in Figure 6.2, which demonstrates in a single take all the Station’s

features numbered as follows:

1. Snapshot Explorer : list of snapshot files under study, organised in a

hierarchical explorer interface.

4See Appendix C for a detailed database schema.

Page 91: AgentSpotter: a MAS Profiling System for Agent Factory

6.6. AgentSpotter Visualisation Components 77

2. Summary Tables : flat profile listings.

3. Call Graph Tree View : hierarchical profile visualisation.

4. Space-Time Diagram: comprehensive performance behaviour visuali-

sation.

5. Bird’s Eye View : navigation mini-map window for the focused Space-

Time Diagram.

AgentSpotter Station was assembled as a standalone modular application

using the Netbeans Platform APIs, the same “rich client” framework that

powers the Netbeans IDE [33]. The main reason behind this choice was the

preexistence of the Agent Factory Netbeans IDE plugin. It seemed logical

to use the same architecture to build the profiler standalone interface with

the aim of converting it later to a set of plugins.

Netbeans Platform’s most spectacular feature is the provision of a versa-

tile window layout manager which supports the composition of sophisticated

views similar to the screen shot shown in Figure 6.2. As previously explained

in §5.4.2, a flexible window layout system not only enhances the user ex-

perience, but also expands the user’s analysis possibilities by facilitating

side-by-side comparisons.

While the flat profile view implementation was trivial and does not

warrant an extended discussion — the summary tables are simple listings,

the implementation of the graphical user interfaces was a more complex task

which we describe in the next section.

6.6 AgentSpotter Visualisation Components

To promote modularity and reuse, we have designed the AgentSpotter Call

Graph Tree View and Space-Time Diagram visualisations as self-contained

Page 92: AgentSpotter: a MAS Profiling System for Agent Factory

78 AgentSpotter design and implementationF

igure

6.2:A

gent

Sp

otterStation

sample

screensh

otsh

owin

gall

the

available

view

sin

asin

gletake

Page 93: AgentSpotter: a MAS Profiling System for Agent Factory

6.6. AgentSpotter Visualisation Components 79

and reusable graphical components. In addition, following good engineering

practices, we did not develop these visualisations from scratch using raw

Java2D API calls. Instead, we have utilised two leading graphical toolkits

particularly suited to the development of interactive zoomable user interfaces:

prefuse [18] and Piccolo2D [1]. The resulting Visualisation Components

library can be reused in any Java Swing application, not just AgentSpotter

Station.

6.6.1 Call Graph Tree View

Figure 6.3: Call Graph Tree View screen shot

In order to construct our Call Graph Tree View, we have adapted a special

tree display component included in prefuse toolkit. We have essentially

managed to implement all the features described in §5.6, including the

keyword search facility which highlights nodes containing a given keyword.

A sample screen shot of the resulting display can be seen in Figure 6.3.

Page 94: AgentSpotter: a MAS Profiling System for Agent Factory

80 AgentSpotter design and implementation

Screen shot comment

The element numbered 1 on the screen shot is the tree root, i.e. the session

level which represents 100% of the recorded activity time. The tree root

is highlighted in blue because it is the current tree selection. As such, it

determines the branch that is expanded; as stated in §5.6.5 the idea is

to reduce the visual clutter. The tree expansion algorithm ensures that

all the children of a selected node are visible as well as its grandchildren.

Consequently, when the tree root is selected, only the first two subsequent

levels are expanded, that is the emitter agent level, and the receiver agent

level. Hence, selecting an emitter agent node should make the FIPA ACL

message level visible, and so on. As an illustration, the call graph numbered

3 shown in Figure 6.2 screen shot, has an agent receiver node selected; as a

result, this branch is fully expanded down to the message impact level.

The element numbered 2 is a text area used to enter a search keyword.

The number of nodes matching the keyword is displayed and all the matching

nodes that are visible are highlighted in pink, just like the element numbered

3, which is one of the nine nodes containing the “explorer” keyword. In a

large expanded tree, the pink nodes really do stand out.

Controls

The view can be controlled using the mouse or the keyboard as follows:

Select: Click an item to expand its tree branch.

Pan: Left-click and drag the background to pan the display view.

Zoom: Right-click and drag the mouse up or down or use the scroll wheel

to zoom the display view.

Zoom-To-Fit: Right-click once to zoom the display to fit the whole tree.

Page 95: AgentSpotter: a MAS Profiling System for Agent Factory

6.6. AgentSpotter Visualisation Components 81

Search: Type in a search query to search over the currently visible nodes.

Layout: Ensure the view window has the focus then press one of the

following layout control keys: Control-1 (left to right tree, the default

one), Control-2 (top to bottom tree), Control-3 (right to left tree) or

Control-4 (bottom to top).

6.6.2 Space-Time Diagram

The screen shot in Figure 6.4 shows a sample Space-Time Diagram View.

In contrast with the Call Graph, this diagram was not adapted from a

pre-existing visualisation component. All the major features specified in §5.8

were included in the beta version except the external platform messaging

display (see §5.8.6). The following paragraphs describe the resulting view as

numbered in Figure 6.4. However, to avoid repeating the contents of §5.8,

only the interesting implementation details are reported here.

1. Session Time Line. This element was implemented exactly as described

in §5.8.1.

2. CPU Load % Line. As specified in §5.8.2, if the system supports CPU

monitoring, for each second, the CPU line plots the current system load

as a gradient area graph. Above the 50% load line, the plotting colour is

gradually shifted from green to red.

3. Agent Life Line. Our sample screen shot (figure 6.4) clearly shows how

the background colours of the life lines vary with the impact of each agent

on the global performance, as stipulated in §5.8.3. Also, the life line captions

“stick” to the left border of the view port so that developers know whose

life line is currently displayed. For time reasons, we didn’t implement the

Page 96: AgentSpotter: a MAS Profiling System for Agent Factory

82 AgentSpotter design and implementation

advanced life line manipulation operations, i.e. moving life lines around and

hiding them selectively.

4. Message Lines. Implemented exactly as stated in §5.8.5.

5. Performance Timed Events. Implemented exactly as specified in §5.8.4.

6. Tooltip Window. This element is mentioned in §5.8.7. In our sample

screen shot (figure 6.4), the tooltip window gives the exact time stamp

associated with a position on the session time line.

7. Message Send/Receive Events. The outgoing mail box represents a

message send event, and the little opened envelope represents a receive

event. To reinforce the message lines visual expressiveness, the icons are

respectively used as origin and destination for the message lines (see §5.8.5).

8. Bird’s Eye View. As made obvious in our sample screen shot (figure 6.4),

the Bird’s Eye View provides important visual information about the current

view port global position whilst enabling developers to navigate quickly

around a large display by dragging the translucent blue rectangle (see §5.8.8).

9. Navigation Scroll Bars. Another way to navigate swiftly around the

diagram is provided by the horizontal and vertical scroll bars.

Controls

Developers can interact with the diagram using the mouse as follows:

Tooltip: Hover the mouse pointer over an item to display a tooltip window.

Pan: Left-click and drag the background to pan the display view.

Page 97: AgentSpotter: a MAS Profiling System for Agent Factory

6.7. Conclusion 83

Zoom: Right-click and drag the mouse left or right to zoom the display

view.

Zoom-To-Fit Item: Left double-click any item to zoom the display to fit

the item.

Zoom-To-Fit Enclosing Item: Right double-click any item to zoom the

display to fit the enclosing item.

Scroll: Use the scroll bar controls to scroll the display view.

Navigate: Left-click and drag the translucent blue rectangle in the Bird’s

Eye View to change the current view’s focus.

6.7 Conclusion

To sum up, we have documented the implementation details of the Agent-

Spotter profiler system for Agent Factory. We have shown that we have

managed to cover 95% of the required feature set in the beta version of

AgentSpotter: only the multi-platform aspect was left out.

From an engineering point of view, the main lesson of the AgentSpotter

system development is the high level of reuse that can be achieved nowadays,

thanks to the wide availability of high quality free libraries and frameworks

in every functional domain. We have, in a very short time, successfully

produced a beta level application, not a prototype.

From a functional point of view, our program has proved it provides

a superior user experience. Admittedly, user experience should not be a

primary concern in a standard academic software project. In the domain of

interactive visualisation programs however, user experience is the essential

element that transforms a tool into an extension of a developer’s mind,

Page 98: AgentSpotter: a MAS Profiling System for Agent Factory

84 AgentSpotter design and implementation

Figure 6.4: Space-Time Diagram screen shot. Obtained from the AgentHerders application [14].

unobtrusively unlocking the access to areas or dimensions that are otherwise

hard or even impossible to reach. Developing multiagent systems is a complex

task, therefore MAS developers need smart tools to tame this complexity.

In the next chapter, with the intention of validating our approach, we

have evaluated the effectiveness of AgentSpotter on a set of representative

Agent Factory applications.

Page 99: AgentSpotter: a MAS Profiling System for Agent Factory

CHAPTER 7Agent Spotting: an evaluation

7.1 Introduction

In order to validate AgentSpotter, we have used it to profile a set of repre-

sentative Agent Factory applications, namely:

• AFBenchmark: a benchmarking application that we have created

specifically for AgentSpotter.

• Agent Sherpa: a simulation designed by fellow students Iain Hull,

Chris Walsh and Aidan Morrissey [21].

• Agent Herders: a complex program that was entered to the Multi-

Agent Programming Contest 2008 by the Dublin Bogtrotters [14].

The goal of our evaluation was threefold:

• To ensure that the program works as expected.

• To demonstrate that AgentSpotter really provides unprecedented

insights into agent-oriented applications.

85

Page 100: AgentSpotter: a MAS Profiling System for Agent Factory

86 Agent Spotting: an evaluation

• To verify AgentSpotter’s ability to cope with high volume and complex

profiling data.

7.2 Benchmark application

7.2.1 Specification

While developing AgentSpotter, we had to create a specific testbed Agent

Factory application that could exercise the whole feature set. The require-

ments for this application can be summarised as follows:

• Load history: a normally distributed random load history should be

generated so that we can get an idea of a “normal” profile which can

be contrasted with “abnormal” profiles where, for example, a single

agent is monopolising all the load, or the load is spread equally among

all agents.

• Agents: the number of active agents should be changeable dynamically

to simulate process escalation.

• Interactions: in addition to direct interactions, the application should

exercise some task delegation scenarios. The idea is to generate

multiple hops messaging scenarios and see their impact on performance.

• Messages: agents should generate a steady flow of messages with

occasional bursts of intense communication.

• Performance events: all three performance behaviours described in

§5.8.4 should be represented, i.e. green (t ≤ 50% time slice), orange

(50% ≤ t ≤ 75% time slice), and red (t > 100%).

We managed to fulfil these requirements by creating a MAS with overseer

agents that request worker agents to execute small, medium or large tasks.

Page 101: AgentSpotter: a MAS Profiling System for Agent Factory

7.2. Benchmark application 87

Worker agents that have been recently overloaded will simply refuse to carry

out the tasks (in a real application they would inform back the requester

about their refusal). From time to time, overseer agents would request

agents to delegate some tasks. In this case, worker agents will behave as

overseers just for one round.

In terms of user interface, the benchmark application is not very spec-

tacular. It only offers a control window shown in Figure 7.1 which allows

developers to start and pause the process, with a slider control that sets the

number of active worker agents.

7.2.2 Evaluation scenario and objective

In effect, the benchmark application has allowed us to test the following

scenario:

• Start the session with a dozen of worker agents and two overseer

agents.

• After 10 minutes add 15 worker agents to spread the load.

• After 4 minutes, suspend the process for 20 seconds.

• Reduce back the number of worker agents to a dozen.

• Run for 5 minutes more then stop the session.

The objective of the performance analysis is twofold: to see the effect

of increasing the number of worker agents on a profile, and to see how

AgentSpotter deals with a relatively long running session.

7.2.3 Flat profile

The resulting flat profile of this test is reproduced in Table 7.1. For the

reader’s convenience, the maximum value for each column is identified by

Page 102: AgentSpotter: a MAS Profiling System for Agent Factory

88 Agent Spotting: an evaluation

an enclosing box. Overseer agents are called “master1” and “master2”. The

worker agents are called “agent” followed by a number e.g. “agent007”.

Firstly, the benchmark appears to make a good job of producing a load

history following a normal distribution1.

Secondly, we can draw the following conclusions from a quick study

of Table 7.1:

• The most active agents in terms of number of iterations are the overseer

agents, “master1” and “master2”, however in terms of CPU load and

overload, three worker agents are topping the list with 30% of the

total activity: “agent001”, “agent009”, and “agent003”.

• The agents with the highest CPU load also display a high number of

time slice overshoots, and a high average time slice duration.

• As expected, the overseer agents were very busy exchanging messages

with the workers. But it seems that messaging is not CPU intensive2,

so their activity percentage is very low, at 1% only. In other words,

bossing other agents around is not really tiring!

In the end, as the performance is distributed normally, the flat profile

does not reveal that much, it just confirms with numbers our original design

principles. Which is, of course, exactly the result we expected.

1Applying the Shapiro-Wilk test for normality on the activity percentage columnyields W = 0.81 with p < 0.0001. The closer to 1 W , the more probable our nullhypothesis that the data set comes from a normally distributed population.

2However, it would be really interesting to get an idea of the Message TransportService load by adding a special performance data channel.

Page 103: AgentSpotter: a MAS Profiling System for Agent Factory

7.2. Benchmark application 89

7.2.4 Call graph and space-time diagram

In contrast, the visualisations provide far more intriguing insights. For a

start, the session call graph3 indicates that messages sent by “master1” have

induced 74.3% of the session activity, whereas “master2” has commandeered

only 20.5%. To put it differently, “master1” was a more effective boss than

“master2”. Unsurprisingly, the agents that “master1” had the most impact

on were: “agent001” (10.1%), “agent009” (7.3%), and “agent003’ (6.2%).

These agents are of course the top performers revealed by the flat profile.

Both overseers were running the same code, and even shared the same

random seeds, so why 70/30 instead of the intuitive 50/50?

Maybe a look at the space-time diagram displayed in Figure 7.2 will

help us unravel this mystery. The numbered areas on this diagram can be

described as follows:

1. This portion of the diagram shows what happens when only 12 workers

are active. The large red rectangles that usually come in groups of

three represent the time-consuming tasks ordered by the overseer

agents. These blocks never overlap because of the way Agent Factory

schedules agents4, and they come in bursts, because both overseers

send the same order to the same worker at the same time5.

2. At this point, 15 more workers are added to the fray, after a slight

pause revealed by the absence of message lines.

3We did not provide a screen shot for the call graph due to its size: the resolutionloss from scaling down a bitmap would have made the text illegible.

4The Agent Factory scheduler does not preempt agents that have exceeded their timeallocation.

5This was revealed by zooming into what appeared to be a single message line. At ahigh magnification level, there were in fact two messages lines within a few microsecondsinterval to the same worker.

Page 104: AgentSpotter: a MAS Profiling System for Agent Factory

90 Agent Spotting: an evaluation

3. This third portion shows the impact of the new workers. The red

blocks are still there, but they are better spread among the agents, so

they don’t come as clumps of three anymore. It looks as if the system

has successfully scaled up.

4. The Bird’s Eye View reveals the bigger picture, and reminds us that

we are looking only at one third of the overall session.

7.2.5 Conclusion

There would be much more to say, but we have clearly demonstrated the

explaining power of AgentSpotter. Unfortunately, we didn’t solve the 70/30

overseer time sharing mystery.

After obtaining these very positive results within a carefully controlled

environment, we were motivated to try agent spotting applications in the

“wild”, as exposed in the following sections.

Figure 7.1: Benchmark application control window

7.3 Agent Sherpa

Agent Sherpa is a simulation of a system “surveying hostile environments

with remote patrol agents” [21]. It was written by fellow students as an

assignment for an agent-oriented course we have taken part in. We have

Page 105: AgentSpotter: a MAS Profiling System for Agent Factory

7.3. Agent Sherpa 91

Table 7.1: Benchmark application flat profile

Total Session Time 18:50.691Total Activity 10:29.164Messages Sent 1206Messages Received 1206Time Slice Duration 1000 ms

Agent T > 0 T > 100% Activity % Session Max(T ) Average(T ) Msg. Msg.iterations overload mm:ss.ms activity ss.ms ss.ms sent rec.

agent001 338 22 1:08.564 10.90 3.740 0.202 6 57agent009 365 21 1:04.257 10.21 3.425 0.176 13 77

agent004 349 22 1:01.529 9.78 3.235 0.176 10 69agent014 284 14 46.413 7.38 3.148 0.163 2 36agent003 401 13 43.881 6.97 3.323 0.109 12 76agent006 361 12 40.141 6.38 3.279 0.111 12 73agent005 367 12 34.903 5.55 3.325 0.095 17 76agent013 301 9 34.716 5.52 3.190 0.115 14 71agent007 378 11 31.864 5.06 3.356 0.084 21 71agent008 357 7 30.850 4.90 3.201 0.086 14 72agent010 330 8 30.280 4.81 3.147 0.091 21 81agent015 285 9 29.382 4.67 3.257 0.103 4 42agent002 348 8 23.196 3.69 3.147 0.066 9 70agent011 357 5 19.363 3.08 3.095 0.054 4 39agent012 225 3 13.172 2.09 3.049 0.058 9 41

master2 901 0 6.681 1.06 0.183 0.007 504 86

master1 873 0 6.485 1.03 0.227 0.007 514 82agent024 46 2 6.281 1.00 3.045 0.136 3 7agent019 31 1 4.449 0.71 3.014 0.143 0 5agent026 42 1 4.400 0.70 3.084 0.104 0 4agent030 26 1 4.002 0.64 3.132 0.153 2 8agent017 46 1 3.811 0.61 3.031 0.082 0 3agent025 40 1 3.767 0.60 3.006 0.094 0 3agent027 31 1 3.694 0.59 3.103 0.119 0 2agent018 38 1 3.384 0.54 3.044 0.089 2 7agent020 39 0 1.762 0.28 0.547 0.045 0 3agent022 47 0 1.523 0.24 0.559 0.032 5 13agent021 32 0 1.300 0.21 0.555 0.040 2 7agent016 219 0 1.194 0.19 0.555 0.005 2 6agent029 38 0 1.039 0.17 0.550 0.027 2 8agent028 45 0 0.749 0.12 0.546 0.016 1 4agent032 34 0 0.749 0.12 0.561 0.022 1 4agent031 36 0 0.742 0.12 0.545 0.020 0 2agent023 40 0 0.598 0.10 0.543 0.014 0 1agent033 30 0 0.043 0.01 0.003 0.001 0 0

Page 106: AgentSpotter: a MAS Profiling System for Agent Factory

92 Agent Spotting: an evaluation

Figu

re7.2:

Ben

chm

arkap

plication

sample

space-tim

ediagram

(18m

inute

long

session)

Page 107: AgentSpotter: a MAS Profiling System for Agent Factory

7.3. Agent Sherpa 93

selected Agent Sherpa because it explicitly endorses a “Command & Control”

architecture where a central unit “monitors the drones and predicts locations

of other IED’s6 or ‘hotspots’ of enemy activity.” [21]

Such a centralised architecture is a potential problem for scaling up, as

we have explained in Table 4.1, Chapter 4. Hence our interest in profiling

a first run with 3 drones only, and then a second run with 12 drones, in a

view to study the scaling behaviour of the system.

The 3-drone session profile listed in Table 7.2 confirms the centralisation

of the application. The “CandC” agent represents 70.83% of the computation

time. Despite 7 time slice overshoots, the system performs well, and has

efficiently executed its assigned task. The call graph numbered 2 displayed

in Figure 7.3 reveals that outgoing messages from “drone2” have the highest

impact because they determine 74.2% of the overall computation time. This

impact consists entirely in “CandC” processing map data using sensor data

returned by the drone. As shown in the call graph, map data is returned

as “mapData(...)” using the inform performative. Conversely, outgoing

messages from “CandC” account for 25.8% of the total computation time.

This impact is distributed among the drones broadly the same way it is

distributed in the profile.

A quick study of the space-time diagram, numbered 2 in Figure 7.4, first

shows that the process is rather CPU intensive (red gradient throughout),

but also reveals the existence of a regular rhythm in the execution of the

system. The very dark “CandC” life line is the visual expression of the

agent’s monopoly on computation time.

The second session, running 12 fully active drones, has a strikingly

different space-time diagram, numbered 1 in Figure 7.4. First the “CandC”

6Stands for “Improvised Explosive Devices”.

Page 108: AgentSpotter: a MAS Profiling System for Agent Factory

94 Agent Spotting: an evaluation

life line is nearly black, which is confirmed by the 96% computation time

share in the flat profile listed in Table 7.3. Then the “CandC” life line starts

with an uninterrupted series of green boxes, which turn completely red after

one minute of run-time, and end up saturating the life line. Clearly, as

confirmed by the CPU usage line, the system is on its knees. A quick glance

at the call graph numbered 1 in Figure 7.3 shows that the outgoing messages

from “drone1” have determined 93.3% of the total computation time! Does

that mean that “drone1” is the culprit?

Since we only aimed to show that AgentSpotter can help discover prob-

lems, we won’t really try to solve the questions raised by the Agent Sherpa

profiles study7. However, we have confirmed the effectiveness of Agent-

Spotter tools on a “real” application. In the next section we will see how

AgentSpotter behaves when confronted with a much more complex MAS.

Table 7.2: Agent Sherpa application flat profile for 3 drone session

Total Session Time 2:49.363Total Activity 25.367Messages Sent 320Messages Received 320Time Slice Duration 300 ms

Agent T > 0 T > 100% Activity % Session Max(T ) Average(T ) Msg. Msg.iterations overload mm:ss.ms activity ss.ms ss.ms sent rec.

CandC 177 7 17.968 70.83 0.623 0.101 237 83drone3 419 1 3.467 13.67 0.354 0.008 37 79drone2 424 0 2.753 10.85 0.264 0.006 37 79drone1 220 0 1.179 4.65 0.140 0.005 9 79

7Our theory is that “CandC” is overwhelmed by the number of requests sent back byall the drones at the same time.

Page 109: AgentSpotter: a MAS Profiling System for Agent Factory

7.3. Agent Sherpa 95

Fig

ure

7.3:

Com

par

ison

ofA

gent

Sher

pa

call

grap

hs

Page 110: AgentSpotter: a MAS Profiling System for Agent Factory

96 Agent Spotting: an evaluation

Figu

re7.4:

Com

parison

ofA

gent

Sherp

asp

ace-time

diagram

s–

1:12

concu

rrent

dron

es;2:

3con

curren

tdron

es

Page 111: AgentSpotter: a MAS Profiling System for Agent Factory

7.4. Agent Herders 97

Table 7.3: Agent Sherpa application flat profile for 12 drone session

Total Session Time 5:42.244Total Activity 4:58.726Messages Sent 372Messages Received 372Time Slice Duration 300 ms

Agent T > 0 T > 100% Activity % Session Max(T ) Average(T ) Msg. Msg.iterations overload mm:ss.ms activity ss.ms ss.ms sent rec.

CandC 627 421 4:47.707 96.31 1.074 0.458 244 128drone1 202 1 2.104 0.70 1.058 0.010 14 20drone7 225 1 1.363 0.46 0.308 0.006 14 20drone6 228 0 1.292 0.43 0.132 0.005 14 20drone12 167 0 1.017 0.34 0.096 0.006 14 20drone9 136 0 0.995 0.33 0.162 0.007 14 22drone2 142 0 0.804 0.27 0.166 0.005 14 20drone4 160 0 0.661 0.22 0.128 0.004 14 21drone11 95 0 0.658 0.22 0.109 0.006 8 20drone8 115 0 0.588 0.20 0.097 0.005 8 20drone3 75 1 0.568 0.19 0.308 0.007 3 20drone10 121 0 0.521 0.17 0.070 0.004 3 21drone5 109 0 0.448 0.15 0.077 0.004 8 20

7.4 Agent Herders

Agent Herders is a complex application that we won’t try to describe fully

here. In a few words, two teams of 6 agent herders compete in herding cows

on a two-dimensional board using different strategies. We invite the reader

to refer to [14] for a more extensive discussion.

What interests us in Agent Herders is the complexity of its coordination

structure, namely a system of auctions to share tasks between the herd

leader, the map explorer and the herder agents. The comparison of the

different AgentSpotter profiles in Table 7.5 indicates that Agent Herders is an

order of magnitude more complex than the other profiled MASs: thousands

of interactions and activities versus hundreds in around 5 minutes.

Our main goal is to evaluate the performance of AgentSpotter when

Page 112: AgentSpotter: a MAS Profiling System for Agent Factory

98 Agent Spotting: an evaluation

loading the profile of a complex application. The comparison of visualisa-

tions load times in Table 7.6 provides another evidence of Agent Herders’

complexity: it takes AgentSpotter 16× more time to load the call graph for

Herders than to load the call graph for Benchmark! Given that it should

require at least K ×Nk ×Ma×Ωm operations8 to build Herders’ call graph,

it is quite a good performance (see §5.6.3 in Chapter 5 for a discussion about

the call graph model).

The time taken by the space-time diagram is more linear9, therefore it

takes only 3× the time to load it for Herders than for Benchmark.

For the sake of completeness, we have reproduced the Agent Herders’

flat profile in Table 7.4, and call graph in Figure 7.5, but we will leave them

uncommented. We have also taken a screen shot of the space-time diagram

to support an earlier discussion in Figure 6.4 on page 84.

7.5 Evaluation summary

To summarise, we have conducted a thorough evaluation of AgentSpotter

by using it to profile a benchmark application of our creation, a “real” MAS

with scalability issues, and a very complex system that was entered in a

research contest. Each evaluation has confirmed the effectiveness of our tool

in different areas. Firstly, our tool suite works exactly as we have designed

it, and correctly produces all the results we have specified, whatever the

application profiled. Secondly, we have successfully demonstrated that the

novel analysis tools we have created truly provide new insights into the

8Based on the following approximations: K = 9, Nk = 9, Ma = 4349/9, Ωm = 3, ittook around 5.7× 107 operations, i.e. 5.7× 107 ÷ 80secs = 7.1× 105/sec. Of course thisdoes not stand for a rigorous algorithmic analysis, but it gives a good idea of the order ofmagnitude involved.

9Most of the complexity of the space-time diagram lies in the setup of the graphicalscene. Indeed, each small element of the scene requires a Java object. Consequently, alarge diagram requires a large amount of memory.

Page 113: AgentSpotter: a MAS Profiling System for Agent Factory

7.5. Evaluation summary 99

Table 7.4: Agent Herders’ flat profile

Total Session Time 4:40.069Total Activity 1:44.822Messages Sent 4349Messages Received 9806Time Slice Duration 2000 ms

Agent T > 0 T > 100% Activity % Session Max(T ) Average(T ) Msg. Msg.iterations overload mm:ss.ms activity ss.ms ss.ms sent rec.

explorer 768 0 36.099 34.44 1.037 0.047 1515 1562

herdLeader 1543 0 23.736 22.64 0.329 0.015 275 994botagent2 780 0 8.871 8.46 0.238 0.011 547 1095botagent4 765 0 7.992 7.62 0.268 0.010 489 1062botagent5 717 0 7.425 7.08 0.324 0.010 442 1068botagent6 755 0 6.954 6.63 0.284 0.009 465 1063botagent1 700 0 6.354 6.06 0.266 0.009 307 1055botagent3 658 0 5.531 5.28 0.361 0.008 309 997hManager 375 0 1.860 1.77 0.320 0.004 0 910

Table 7.5: Comparison of the profiled sessions in terms of number of events

Event type Herders Sherpa-12 Sherpa-3 Benchmark

agentContainer/resume 9 13 4 105agentContainer/suspend 0 0 0 22cpu 280 342 169 1130message/receive 9806 372 320 1206message/send 4349 372 320 1206scheduler/activity 7061 2402 1240 7680

Session time 4:40.069 5:42.244 2:49.363 18:50.691

Page 114: AgentSpotter: a MAS Profiling System for Agent Factory

100 Agent Spotting: an evaluation

Figu

re7.5:

Agen

tH

erders’

callgrap

h

Page 115: AgentSpotter: a MAS Profiling System for Agent Factory

7.5. Evaluation summary 101

Table 7.6: Comparison of AgentSpotter visualisations load times on a 2GHzDual Core Pentium machine

Agent Herders Benchmark

Flat profile < 1 sec < 1 secSpace-time diagram 15 secs 6 secsCall graph 1 min 20 secs 5 secs

behaviour of a MAS. Finally, we have verified the ability of our system to

handle high volume and complex profiles.

Page 116: AgentSpotter: a MAS Profiling System for Agent Factory
Page 117: AgentSpotter: a MAS Profiling System for Agent Factory

CHAPTER 8Conclusions

8.1 Related work

To the best of our knowledge, AgentSpotter is the first profiler system

specifically designed for a MAS using agent-oriented performance concepts,

although there is a strong MAS research history in the main domains covered

by AgentSpotter: MAS developer tools, and MAS visualisation tools.

In the domain of interactive tools specifically geared towards MAS

developers, we can cite, for example, other tools developed for Agent Factory:

the AF Debugger [7], and the VIPER agent protocol editor tool [41].

A fundamental approach on visualisation and debugging tools architec-

ture for MASs can be found in [35] where Ndumu et al. state that visualising

MASs is a “notoriously difficult task” because of the information overload,

and present a system comprising a suite of tools, each tool providing a

different perspective on the studied MAS.

Common techniques used for MAS visualisation are based on clustering

and classification algorithms as in [42] where Shroeder and Noy compare the

relative merits of data dimension reduction techniques in order to generate

103

Page 118: AgentSpotter: a MAS Profiling System for Agent Factory

104 Conclusions

summary visualisations. In contrast, AgentSpotter stresses the importance

of exploiting the human visual processing abilities.

Lam and Barber correctly argue in [27] that “software comprehension ...

is essential for developing, maintaining, and improving software”, notably

MASs. Their global approach, which combines a methodology (Tracing

Method) and a tool (Tracer Tool), has inspired the design of AgentSpotter.

As mentioned in Chapter 4, while MAS performance study is well repre-

sented in the literature, only the Cougaar system [19] offers a performance-

minded infrastructure.

As Lynch and Rajendran remind us in [32], the ultimate goal is “breaking

into industry”, which requires the creation of industry-grade development

tools prefigured by AgentSpotter.

AOSE, and a fortiori MAS profiling, is ultimately just one aspect of

the bigger roadmap for agent-oriented technologies defined in [31] by the

AgentLink organisation.

Outside the MAS domain, AgentSpotter must be related with the recent

explosion of infovis (information visualisation) research, and more specif-

ically the domain of performance visualisation in parallel and distributed

applications as described in [43]. Graph visualisations like the Call Graph

are surveyed by Herman and al. in [20]. The Paje visualisation system

described in [36], developed by France Telecom researchers, is very close to

AgentSpotter in its architecture and visualisation offer, but is targeted at

traditional distributed systems.

8.2 Future work

We envision four main axes of improvement for AgentSpotter.

Firstly, we would like to port our tool to another major Java-based MAS

Page 119: AgentSpotter: a MAS Profiling System for Agent Factory

8.3. General conclusion 105

platform like JADE. As we explain in Chapter 5, only the data capture

apparatus should require a specific implementation.

Secondly, we should expand the range of data captured and displayed by

AgentSpotter. For instance, the performance of additional system services

should be recorded, and more details should be collected about agents’

performance events like the distribution of the execution time among the

sensors, actuators, reasoning engine etc.

The third task would consist in extending AgentSpotter to a multi-plat-

form profiling tool as we suggest in §5.8.6.

The last, and more practical, task would be to make the AgentSpotter

tool even more usable by providing, for example, a way to save pictures and

tables to files, and to execute custom queries on the underlying performance

data1.

8.3 General conclusion

In this dissertation, we have presented our implementation of AgentSpotter

for Agent Factory. It is the first generic MAS profiling system destined to a

Java-based, FIPA compliant, agent-oriented framework, and provides the

following features:

• Mapping of the conventional profiling concepts to agent-oriented con-

cepts.

• Platform-independent architecture.

• Integration of the performance analysis infrastructure right into Agent

Factory’s run-time environment.

1This is already possible outside AgentSpotter by using one of the existing querytools for SQLite databases.

Page 120: AgentSpotter: a MAS Profiling System for Agent Factory

106 Conclusions

• Implementation of a suite of novel interactive visual tools that aid

MAS developers understand and debug performance issues.

We have also successfully demonstrated how AgentSpotter sheds new

light on MAS performance behaviour, by using it to evaluate a set of

representative systems designed with Agent Factory.

It is our hope that this contribution, which facilitates the conduct of

MAS performance analysis, will foster more interest in the comparative

performance studies of MAS designs.

Page 121: AgentSpotter: a MAS Profiling System for Agent Factory

Appendices

107

Page 122: AgentSpotter: a MAS Profiling System for Agent Factory
Page 123: AgentSpotter: a MAS Profiling System for Agent Factory

Appendix A

Building AgentSpotter from the subversion

repository

AgentSpotter is available from the Agent Factory (AF) area on SourceForge

at http://sourceforge.net/projects/agentfactory. It is released un-

der the same licence as AF, i.e. the GNU Lesser General Public License,

and comes with full source code.

If you wish to build it from source, here is the procedure to follow. First

ensure that your system meets the following requirements:

• Netbeans IDE version 6.1.

• JDK version 1.6+.

Then download the GNU tarball from http://agentfactory.svn.sourceforge.

net/viewvc/agentfactory/agentspotter/trunk/, or checkout the project’s

trunk using SVN by using the following command-line:

svn co https : // agent f a c to ry . svn . s o u r c e f o r g e . net / svnroot /

agent f a c to ry / agent spo t t e r / trunk agent spo t t e r

Open the Netbeans IDE, then open the src/AgentSpotterStation

directory as a project (File→Open Project... menu). Ensure that the “Open

Required Projects” check box in the displayed “Open Project” dialog box is

checked, as shown on Figure A.1.

109

Page 124: AgentSpotter: a MAS Profiling System for Agent Factory

110 Building AgentSpotter from the subversion repository

Figure A.1: Netbeans “Open Project” dialog box

Once the project is opened use the usual Netbeans commands to build

the application.

Page 125: AgentSpotter: a MAS Profiling System for Agent Factory

Appendix B

Getting started with AgentSpotter

B.1 Profiling

Here are the steps required to profile an existing AF application:

• First ensure that your application is compatible with AF version 1.2.

• Also, if your application has a main frame, verify that the default

close operation is to exit the application i.e. add the following code if

needed:

. . .

JFrame mainFrame = new MyFrame( ) ;

mainFrame . s e tDe fau l tC lo seOperat ion (

WindowConstants .EXIT ON CLOSE) ;

. . .

The idea is to make sure that the application shuts down gracefully so

that AgentSpotter has a chance to flush its database records.

• Add the following line to the platform configuration file (.CFG):

PLATFORMGUI com.agen t f a c to ry . agen t spo t t e r .AgentSpot t e r

Alternatively, you can specify:

PLATFORMGUI com.agen t f a c to ry . agen t spo t t e r .AgentSpot t e r

AUTOCOMMIT

111

Page 126: AgentSpotter: a MAS Profiling System for Agent Factory

112 Getting started with AgentSpotter

This will flush the database at each write, which is rather slow but is

necessary if the MAS has not determined exit point.

• Assuming that you have downloaded the distribution from http:

//sourceforge.net/projects/agentfactory and that you have un-

compressed it to a directory called agentspotter, add all the JAR

files from agentspotter/lib to the application class path.

• Run the MAS.

• When the MAS has terminated, load the generated “profile.aspot” file

into AgentSpotter Station. This file is generated in the default current

directory of the profiled AF application.

B.2 AgentSpotter Station

The AgentSpotter Station GUI is rather intuitive and tries to follow the

user interface conventions used by the Netbeans IDE. Here are a few points

that will help you getting started:

• To load a snapshot file, right click on the root element titled “Agent-

Spotter Snapshots” in Snapshot Window (element numbered 1 shown

in Figure 6.2 screen shot on page 78), and select the Add Snapshot

File... in the contextual menu.

• In the displayed open file dialog, locate and load the “profile.aspot”

file you have just generated.

• If any window is missing from the main display, you can make it appear

by going to the Window menu and selecting the appropriate menu

entry i.e. Snapshot Window or Bird’s Eye Window.

Page 127: AgentSpotter: a MAS Profiling System for Agent Factory

B.2. AgentSpotter Station 113

• To display a visualisation, expand a snapshot file loaded in the Snapshot

Window, then right click on a session to pop up a contextual menu,

then select the desired visualisation.

• Check the Netbeans documentation for an explanation of how the

window layout system works.

Page 128: AgentSpotter: a MAS Profiling System for Agent Factory
Page 129: AgentSpotter: a MAS Profiling System for Agent Factory

Appendix C

AgentSpotter Snapshot File Database Schema

Figure C.1: Database schema diagram of the AgentSpotter profile data file

115

Page 130: AgentSpotter: a MAS Profiling System for Agent Factory
Page 131: AgentSpotter: a MAS Profiling System for Agent Factory

Bibliography

[1] Benjamin B. Bederson, Jesse Grosjean, and Jon Meyer. Toolkit design

for interactive structured graphics. IEEE Transactions on Software

Engineering, 30:535–546, 2004.

[2] Fabio Luigi Bellifemine, Giovanni Caire, and Dominic Greenwood.

Developing Multi-Agent Systems with JADE. Wiley Series in Agent

Technology. Wiley, 2 2007.

[3] B. Brassel, M. Hanus, F. Huch, J. Silva, and G. Vidal. Run-time

profiling of functional logic programs. In Proceedings of the Interna-

tional Symposium on Logic-based Program Synthesis and Transforma-

tion (LOPSTR04), pages 182–197. Springer LNCS, 2005.

[4] F. Brazier, M. van Steen, and N. Wijngaards. On MAS scalability. In

T. Wagner and O. Rana, editors, Proceedings of Second International

Workshop on Infrastructure for Agents, MAS, and Scalable MAS, pages

121–126, 05 2001.

[5] Bradley J. Clement and Anthony C. Barrett. Continual coordination

through shared activities. In AAMAS ’03: Proceedings of the second

international joint conference on Autonomous agents and multiagent

systems, pages 57–64, New York, NY, USA, 2003. ACM.

117

Page 132: AgentSpotter: a MAS Profiling System for Agent Factory

118 Bibliography

[6] R. W. Collier, M. J. O’Grady, G. M. P. O’Hare, C. Muldoon, D. Phelan,

R. Strahan, and Y. Tong. Self-organisation in agent-based mobile

computing. In DEXA ’04: Proceedings of the Database and Expert

Systems Applications, 15th International Workshop, pages 764–768,

Washington, DC, USA, 2004. IEEE Computer Society.

[7] Rem W. Collier. Debugging agents in agent factory. In Rafael H.

Bordini, Mehdi Dastani, Jurgen Dix, and Amal El Fallah-Seghrouchni,

editors, PROMAS, volume 4411 of Lecture Notes in Computer Science,

pages 229–248. Springer, 2006.

[8] Rem W. Collier. Web site for Agent Factory, 2008. http://www.

agentfactory.com/ (accessed October, 2008).

[9] Rem W. Collier. Web site for Agent Factory, Platform Service De-

velopment Guide, 2008. http://www.agentfactory.com/index.php/

Platform_Service_Development_Guide (accessed October, 2008).

[10] Rem W. Collier. Web site for Agent Factory, The Logger API Guide,

2008. http://www.agentfactory.com/index.php/The_Logger_API_

Guide (accessed October, 2008).

[11] Rem W. Collier, Gregory M. P. O’Hare, T. D. Lowen, and Colm

Rooney. Beyond prototyping in the factory of agents. In Vladimır

Marık, Jorg P. Muller, and Michal Pechoucek, editors, Multi-Agent

Systems and Applications III, 3rd International Central and Eastern

European Conference on Multi-Agent Systems, CEEMAS 2003, Prague,

Czech Republic, June 16-18, 2003, Proceedings, volume 2691 of Lecture

Notes in Computer Science, pages 383–393. Springer, 2003.

Page 133: AgentSpotter: a MAS Profiling System for Agent Factory

Bibliography 119

[12] Claudio Cubillos, Franco Guidi-Polanco, and Ricardo Soto. Performance

analysis of a multiagent architecture for passenger transportation. In

Jan van Leeuwen, Giuseppe F. Italiano, Wiebe van der Hoek, Christoph

Meinel, Harald Sack, and Frantisek Plasil, editors, SOFSEM (1), volume

4362 of Lecture Notes in Computer Science, pages 591–600. Springer,

2007.

[13] R. Deters. Scalability and Multi-Agent Systems. In AGENTS ’01: Pro-

ceedings of Workshop “Infrastructure for Scalable Multi-Agent Systems”,

pages 127–134, 2001.

[14] M. Dragone, D. Lillis, C. Muldoon, R. Tynan, R. W. Collier, and

G. M. P. OHare. Dublin bogtrotters: Agent herders. In Proceedings of

the 6th International Workshop on Programming Multi Agent Systems

(PROMAS-2008), Estoril, Portugal, 2008.

[15] FIPA. FIPA ACL Message Structure Specification. FIPA, 2001.

[16] FIPA. FIPA Abstract Architecture Specification. FIPA, 2002.

[17] Susan L. Graham, Peter B. Kessler, and Marshall K. Mckusick. Gprof:

A call graph execution profiler. SIGPLAN Not., 17(6):120–126, 1982.

[18] Jeffrey Heer, Stuart K. Card, and James A. Landay. prefuse: a toolkit

for interactive information visualization. In CHI ’05: Proceeding of

the SIGCHI conference on Human factors in computing systems, pages

421–430, New York, NY, USA, 2005. ACM Press.

[19] Aaron Helsinger, Richard Lazarus, William Wright, and John Zinky.

Tools and techniques for performance measurement of large distributed

Page 134: AgentSpotter: a MAS Profiling System for Agent Factory

120 Bibliography

multiagent systems. In AAMAS ’03: Proceedings of the second interna-

tional joint conference on Autonomous agents and multiagent systems,

pages 843–850, New York, NY, USA, 2003. ACM.

[20] Ivan Herman, Ieee Cs Society, Guy Melanon, and M. Scott Marshall.

Graph visualization and navigation in information visualization: A

survey. IEEE Transactions on Visualization and Computer Graphics,

6:24–43, 2000.

[21] Iain Hull, Chris Walsh, and Aidan Morrissey. Web site for Agent

Sherpa, Surveying Hostile Environments with Remote Patrol Agents,

2008. http://code.google.com/p/agent-sherpa/ (accessed October,

2008).

[22] Hwaci. Web site for the SQLite Database Engine, 2008. http://www.

sqlite.org/ (accessed October, 2008).

[23] IEEE. Web site for FIPA, 2008. http://www.fipa.org/ (accessed

October, 2008).

[24] Zoltan Juhasz and Prasenjit Paul. Scalability Analysis of the Contract

Net Protocol. ccgrid, 0:346, 2002.

[25] Kresimir Jurasovic, Gordan Jezic, and Mario Kusek. A Performance

Analysis of Multi-Agent Systems. ITSSA, 1(4):335–342, 2006.

[26] Donald E. Knuth. An empirical study of FORTRAN programs. j-SPE,

1(2):105–133, April/June 1971.

[27] D. N. Lam and K. S. Barber. Comprehending agent software. In

AAMAS ’05: Proceedings of the fourth international joint conference

Page 135: AgentSpotter: a MAS Profiling System for Agent Factory

Bibliography 121

on Autonomous agents and multiagent systems, pages 586–593, New

York, NY, USA, 2005. ACM.

[28] Leslie Lamport. Time, clocks, and the ordering of events in a distributed

system. Commun. ACM, 21(7):558–565, July 1978.

[29] L. C. Lee, H. S. Nwana, D. T. Ndumu, and P. De Wilde. The stability,

scalability and performance of multi-agent systems. BT Technology

Journal, 16(3):94–103, 1998.

[30] David Lillis, Rem Collier, Fergus Toolan, and John Dunnion. Evaluating

communication strategies in a multi agent information retrieval system.

In Proceedings of the 5th European Workshop on Multi-Agent Systems

(EUMAS’07), Hammamet, Tunisia, December 13-14th 2007.

[31] M. Luck, P. McBurney, O. Shehory, and S. Willmott. Agent Technology:

Computing as Interaction (A Roadmap for Agent Based Computing).

AgentLink, 2005.

[32] Simon Lynch and Keerthi Rajendran. Breaking into industry: tool

support for multiagent systems. In AAMAS ’07: Proceedings of the 6th

international joint conference on Autonomous agents and multiagent

systems, pages 1–3, New York, NY, USA, 2007. ACM.

[33] SUN Microsystems. Web site for Netbeans Platform, 2008. http:

//platform.netbeans.org/ (accessed October, 2008).

[34] Angel Mur, Liu Peng, Rem Collier, David Lillis, Fergus Toolan, and

John Dunnion. A HOTAIR scalability model. In Proceedings of the 16th

Irish Conference on Artificial Intelligence and Cognitive Science (AICS

2005), pages 359–368, Portstewart, Northern Ireland, 2005. University

of Ulster.

Page 136: AgentSpotter: a MAS Profiling System for Agent Factory

122 Bibliography

[35] Divine T. Ndumu, Hyacinth S. Nwana, Lyndon C. Lee, and Jaron C.

Collis. Visualising and debugging distributed multi-agent systems.

In AGENTS ’99: Proceedings of the third annual conference on Au-

tonomous Agents, pages 326–333, New York, NY, USA, 1999. ACM.

[36] F.-G. Ottogalli, Cyril Labbe, V. Olive, Benhur de Oliveira Stein,

Jacques Chassin de Kergommeaux, and Jean-Marc Vincent. Visualisa-

tion of distributed applications for performance debugging. In ICCS

’01: Proceedings of the International Conference on Computational

Science-Part II, pages 831–840, London, UK, 2001. Springer-Verlag.

[37] Parakey Inc. Firebug JavaScript debugger and profiler. http://

getfirebug.com/js.html (accessed August 4th, 2008), 2008.

[38] Liu Peng, Rem Collier, Angel Mur, David Lillis, Fergus Toolan, and

John Dunnion. A self-configuring agent-based document indexing

system. In Proceedings of the 4th International Central and Eastern Eu-

ropean Conference on Multi-Agent Systems (CEEMAS 2005), Budapest,

Hungary, 2005. Springer-Verlag GmbH.

[39] Omer F. Rana and Kate Stout. What is scalability in multi-agent

systems? In AGENTS ’00: Proceedings of the fourth international

conference on Autonomous agents, pages 56–63, New York, NY, USA,

2000. ACM.

[40] W.J. Reed. The Pareto, Zipf and other power laws. Economics Letters,

74(1):15–19, 2001.

[41] C. F. B. Rooney, R. W. Collier, and G. M. P. Ohare. Viper: Visual

protocol editor. In Proceedings of COORDINATION 2004, pages 279–

293. Springer Verlag, 2004.

Page 137: AgentSpotter: a MAS Profiling System for Agent Factory

Bibliography 123

[42] Michael Schroeder and Penny Noy. Multi-agent visualisation based on

multivariate data. In AGENTS ’01: Proceedings of the fifth international

conference on Autonomous agents, pages 85–91, New York, NY, USA,

2001. ACM.

[43] Eric Shaffer, Daniel A. Reed, Shannon Whitmore, and Benjamin Scha-

effer. Virtue: Performance visualization of parallel and distributed

applications. Computer, 32(12):44–51, 1999.

[44] Connie U. Smith. Performance Engineering of Software Systems.

Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA,

1990.

[45] Connie U. Smith and Lloyd G. Williams. Performance solutions: a

practical guide to creating responsive, scalable software. Addison Wesley

Longman Publishing Co., Inc., Redwood City, CA, USA, 2002.

[46] R.G. Smith. The contract net protocol. IEEE Transactions on Com-

puters, 29(12):1104–1113, 1980.

[47] Amitabh Srivastava and Alan Eustace. Atom: a system for building

customized program analysis tools. In PLDI ’94: Proceedings of the

ACM SIGPLAN 1994 conference on Programming language design and

implementation, pages 196–205, New York, NY, USA, 1994. ACM.

[48] Kenneth S. Stephens and Joseph M. Juran. The Non-Pareto Principle—

Mea Culpa. In Juran, Quality, and a Century of Improvement: The

Best On Quality, volume 15, pages 185–190. American Society for

Quality, 2004.

Page 138: AgentSpotter: a MAS Profiling System for Agent Factory

124 Bibliography

[49] Sun Microsystems, Inc. JVM Tool Interface (JVMTI), Version

1.0. Web pages at http://java.sun.com/j2se/1.5.0/docs/guide/

jvmti/ (accessed August 4th, 2008), 2004.

[50] TILAB. Web site for JADE, 2008. http://jade.tilab.com/ (accessed

February 28th, 2008).

[51] Phillip J. Turner and Nicholas R. Jennings. Improving the scalability of

multi-agent systems. In Revised Papers from the International Workshop

on Infrastructure for Multi-Agent Systems, volume 1887, pages 246–262,

London, UK, 2001. Springer-Verlag.

[52] Michael Wooldridge and Nicholas R. Jennings. Intelligent agents: The-

ory and practice. Knowledge Engineering Review, 10:115–152, 1995.

[53] Yahoo! Inc. Speed up your web pages with YSlow. http://developer.

yahoo.com/yslow/ (accessed August 4th, 2008), 2008.