Future Generation Computer Systems 21 (2005) 1192–1198

Oh behave! Agent-based behavioral representationsin problem solving environments

M. North∗, C. Macal, P. Campbell

Argonne National Laboratory, 9700 S. Cass Avenue, Argonne, IL 60439, USA

Available online 12 May 2004

Abstract

The development of deregulated electricity systems around the world has produced the need for simulation systems that arecapable of addressing the complexities that arise in the new markets. Agent-based models allow the use of complex adaptivesystems approaches that are capable of producing tools or problem solving environments that can address the behavior of eachof the participants within the electricity market. The agents in the tools are allowed to establish their own objectives and applytheir own decision rules. They can be developed to learn from their previous experiences and change their behavior when futureopportunities arise. In this paper, we will argue that the same type of agent-based technology that is used to produce “realistic”agent behavior in agent-based simulation tools at Argonne National Laboratory can also be used to embed these tools in problemsolving environments.© 2004 Elsevier B.V. All rights reserved.

Keywords:Problem solving environments; Agent-based modelling; Behavioral representations

1

asiuttct

(

forevenbro-tors.edand

the-ef-rod-has

, andones.m-

pable

0

. Introduction

The development of deregulated electricity systemsround the world has produced the need for simulationystems that are capable of addressing the complex-ties that arise in the new markets. As these electrictility systems continue to evolve from regulated, ver-

ically integrated monopoly structures to open marketshat promote competition among suppliers and provideonsumers with a choice of services, the unbundling ofhe generation, transmission, and distribution functions

∗ Corresponding author.E-mail addresses:north@anl.gov (M. North), macal@anl.gov

C. Macal), campbell@anl.gov (P. Campbell).

that is part of this evolution creates opportunitiesmany new players, or agents, to enter the market. Itcreates new types of industries, including powerkers, marketers, and load aggregators or consolidaAs a result, fully functioning markets are distinguishby the presence of a large number of companiesplayers that are in direct competition. Economicory holds that this will lead to increased economicficiency expressed in higher quality services and pucts at lower retail prices. Each market participantits own unique business strategy, risk preferencedecision model. Decentralized decision-making isof the key features of the new deregulated market

Agent-based models (ABMs) allow the use of coplex adaptive systems approaches that are ca

167-739X/$ – see front matter © 2004 Elsevier B.V. All rights reserved.doi:10.1016/j.future.2004.04.006

M. North et al. / Future Generation Computer Systems 21 (2005) 1192–1198 1193

of producing tools or problem solving environments(PSE) that can address the behavior of each of the par-ticipants within the electricity market. The agents inthe tools are allowed to establish their own objectivesand apply their own decision rules. They can be de-veloped to learn from their previous experiences andchange their behavior when future opportunities arise.

A PSE is a computer system that provides all thecomputational facilities needed to solve a target classof problems. These features include advanced solutionmethods, automatic and semiautomatic selection of so-lution methods, and ways to easily incorporate novelsolution methods. Moreover, PSEs use the language ofthe target class of problems, so users can run them with-out specialized knowledge of the underlying computerhardware or software. By exploiting modern technolo-gies such as interactive color graphics, powerful pro-cessors, and networks of specialized services, PSEs cantrack extended problem solving tasks and allow usersto review them easily. Overall, they create a frameworkthat is all things to all people: they solve simple or com-plex problems, support rapid prototyping or detailedanalysis, and can be used in introductory education orat the frontiers of science[3].

An agent is a software representation of a decision-making unit. Agents are self-directed software objectswith specific traits and typically exhibit bounded ratio-nality, meaning that they make decisions using limitedinternal decision rules that depend only on imperfectlocal information. Emergent behavior is a key featureo av-i ples

thatw asedo on-m aileda fulil tri-c rcesp rva-t el-o lec-t S)m temm latea the

entire system. Rather, agents are allowed to establishtheir own objectives and apply their own decision rules.With its agent-based approach, EMCAS is specificallydesigned to analyze multi-agent markets and allow test-ing of regulatory structures before they are applied toreal systems.

In this paper, we will argue that the same type ofagent-based technology that is used to produce “realis-tic” agent behavior in EMCAS, and other agent-basedsimulation tools at ANL, can also be used to embedthese tools in a PSE-type environment, i.e. one in whichall the intricacies of the underlying computer hardwareand software are hidden from the user, who is then freeto focus on modeling meaningful solutions.

2. EMCAS

EMCAS is an electricity market model related toseveral earlier models[7,8]. EMCAS includes a largenumber of different agents to model the full range oftime scales—from hours to decades—that are neededto understand the domain[6]. The focus of agent rulesin EMCAS varies to match the time continuum, asshown inFig. 1. Over longer time scales, human eco-nomic decisions dominate. Over shorter time scales,physical laws dominate. Many EMCAS agents are rel-atively complex, or “thick”, compared to typical agents.Fig. 1shows nested decision-making time frames. Theshortest time frame, real-time dispatch, involves opera-t conds ailyd thes de-c ithp .

rmd om-p n inF wer,r ationc on re-s man-a utionc verya smis-s temo ents

f ABMs. Emergent behavior occurs when the behor of a system is more complicated than the simum of the behaviors of its components[1].

Many of the modeling tools for systems analysisere developed over the last two decades are bn the implicit assumption of a centralized decisiaking process. Although these tools are very detnd complex and will continue to provide many use

nsights into power systems operation[2,4,5], they areimited in their ability to adequately analyze the inate web of interactions among all the market forevalent in the new markets. Driven by these obse

ions, Argonne National Laboratory (ANL) has devped a new deregulated market analysis tool, the E

ricity Market Complex Adaptive Systems (EMCAodel. Unlike those of conventional electric sysodels, the EMCAS ABM techniques do not postusingle decision maker with a single objective for

ional usage of resources on an hourly basis. The sehortest time frame, decision level 2, consists of decisions made by each of the major agents withinystem. Moving up the time scale implies largerisions being made over longer periods of time wroportionally larger time steps between decisions

EMCAS agents are highly specialized to perfoiverse tasks, ranging from acting as generation canies to modeling transmission lines, as showig. 2. The figure shows consumers that use poegulators that oversee the power system, generompanies (GenCos) that manage power generatiources, transmission companies (TransCos) thatge the long distance transmission system, distribompanies (DisCos) that manage local power delind independent system operators/regional tranion organizations (ISO/RTOs) that coordinate sysperations. To support specialization, EMCAS ag

1194 M. North et al. / Future Generation Computer Systems 21 (2005) 1192–1198

Fig. 1. EMCAS time scales and decision levels.

include large number of highly specific rules. EMCASagent strategies are highly programmable. Users caneasily define new strategies to be used for EMCASagents and then examine the marketplace consequencesof these strategies. EMCAS and its component agents

are currently being subjected to rigorous quantitativevalidation and calibration.

The EMCAS interface client uses Dynamic Hyper-text Markup Language (DHTML) and Scalable Vec-tor Graphics (SVG), allowing it to be displayed in all

Fig. 2. EMCAS structure and agents.

M. North et al. / Future Generation Computer Systems 21 (2005) 1192–1198 1195

major web browsers. The interface client can be usedanywhere in the world that a server is available via theInternet or on portable computers without a networkconnection but with a local server.

To better understand the requirements of an elec-tricity market structure testing tool, a live online elec-tricity market simulation was created. The marketgame that was developed used individuals to playthe role of generation companies. One additional per-son played the role of the ISO/RTO. The generationcompany players bid their available electricity pro-duction into the simulated market based on publicinformation electronically posted by the system op-erator. This information included historical and pro-jected prices, demands, supply, and weather. The sys-tem operator collected the players’ bids on a periodicbasis and used them to simulate the operation of anelectricity spot market. The system prices from a sixplayer market simulation are shown by the dark line inFig. 3.

An EMCAS case has been created based on the pre-viously described market game. The system price re-sults are shown in specific agents representing individ-ual market game players were implemented by usingEMCAS’ agent architecture. The strategies of the indi-vidual players were determined by asking them to writeshort descriptions of their approaches after the comple-tion of the game and then following up the writing witha series of focused interviews. Once the strategies weredetermined, agents implementing each of the strategiesw

The individual agents developed to emulate the mar-ket game players were run using the same data origi-nally used for the game. The system prices from a sixplayer market simulation are shown by the light line inFig. 3. The resulting prices are similar to those foundin the individual market game as shown in the figure.The main difference is that the prices near hour 40 arehigher in the EMCAS case because the EMCAS agentswere programmed to use the evolved final strategies ofthe players. Many of the market game players had be-gun the game using a relatively cautious approach tobidding. As the game progressed, they learned to be-come much more aggressive. The EMCAS agents wereprogrammed with the final, more aggressive strategiesof the human players. Thus, EMCAS tended to havehigher prices throughout the simulation. Once EMCASwas able to replicate the original market game, it wasused to explore its suitability as an electricity marketstructure testing tool.

One agent class used in the EMCAS modeling sys-tem is designed to simulate the behavior of GeneratingCompany Agents (GCAs) and the marketing strategiesthat emerge as GCAs strive to exploit the physical lim-itations of the power system using the market rules un-der which they operate, as shown inFig. 4. GCAs cansell products in various markets. In EMCAS, a GCAlearns the extent to which local and regional prices areinfluenced by its marketing strategies. This learningprocess is based on an “explore and exploit” process.Agents explore various marketing and bidding strate-g ll, it

rices—

ere programmed.

Fig. 3. Market clearing p

ies. Once a strategy is found that performs we

EMCAS vs. market game.

1196 M. North et al. / Future Generation Computer Systems 21 (2005) 1192–1198

Fig. 4. Generation company agent.

is exercised (i.e. exploited) and fine-tuned as subtlechanges occur in the marketplace. When more dramaticmarket changes take place and a strategy begins to fail,an agent more frequently explores new strategies in anattempt to adapt to the dynamic and evolving supply-and-demand forces in the marketplace. Even when astrategy continues to perform well, a GCA periodicallyexplores and evaluates other strategies in its searchfor one that performs better. However, the explorationrate tends to be significantly lower than under stressfulconditions.

In EMCAS, a GCA is comprised of a number ofbuilding blocks that represent different tasks or actionsan agent can perform. Each GCA seeks to arrange andparameterize these building blocks in a way that allowsthe market player to maximize its corporate utility. Abuilding block consists of a set of one or more relativelysimple rules. For example, one very simple agent rulemay be if the GCAs sets the marketing clearing price inthe last bidding period, then the GCA bid price in thenext period will be fractionally higher. One parameterin this building block specifies the rate of change inthe bid price. Although the basic building blocks areavailable to all GCAs, an exploited strategy may notutilize a building block if it is discovered that it is notbeneficial. However, if market conditions change or ifthe GCA discovers a new way to combine the build-ing block with another one, it can be used to developa new strategy. When a GCA owns and operates morethan one generating unit, an integrated strategy is for-

mulated, and the combined effects of unit-level actionsare important. This may entail losing money at one fa-cility to gain more profits at another one.

We propose that this same agent building block ap-proach can be used to develop elements of a PSE. Inparticular, the approach can be adapted to address au-tomatic ontology construction/extension; personaliza-tion; and real-time visual representation of both theprogram state and the “object of interest” state withina PSE.

3. Automatic ontology construction andextension

An EMCAS agent makes decisions based on past ex-periences and anticipated conditions in the future andin the context of current market rules and the potentialimpact that other players will have on the markets. Inthe same way, a PSE environment can be describedwithin a particular domain. The possible/reasonablepaths through the PSE can then be broken down into di-rected graphs of discrete steps, or building blocks, eachcorresponding to a function/action that allows the userto progress towards the goal of “solving” the problem.Analytical agents can then be constructed from the dis-crete steps in response to goals set by the user via theuser interface. As the problem or analysis is workedthrough, the agent reevaluates its context at each step,assembling the necessary blocks as required. An ontol-

M. North et al. / Future Generation Computer Systems 21 (2005) 1192–1198 1197

ogy constructed this way can then be saved for furtheruse in later PSE applications. Since, in use, each stepis accompanied by an evaluation of the blocks neededfor the next step(s), the ontology is extended by sim-ply using the PSE. We have found it possible to buildarbitrarily complex behavior paths using this approach.

3.1. Personalization

By providing a user interface that allows each user toenter their preferences, this initial personalization be-comes part of the PSE environment that the agents au-tomatically use to assemble the problem solving pathsthat are to be used. As the user continues to use thePSE, the agents learn preferred paths, or tool use, bythe continual assessment of the internal PSE environ-ment.

3.2. Real-time visual representations within PSE

True flexibility in a user input-and-display environ-ment can be achieved by having functions delegatethese functions to other services. This can be achievedfor most functions, but is perhaps most easily dis-cussed/illustrated for the case of real-time visual rep-resentation. Meta-protocols have been developed thattransparently link domain objects. This allows domainobjects to publish available data and functionality atrun time, and allows changes in the state of domainobjects to be displayed as they occur.

forf thata jectso playo ingt hichs t thes nt inu lso bed

4

msa PSEst antsw that

the same type of agent-based technology that is usedto produce “realistic” agent behavior in EMCAS andother agent-based simulation tools at ANL can also beused to embed these tools in a PSE-type environment.Furthermore, PSEs are ideally suited to support the in-tense interaction and rapid feedback required for ABMdesign, development and deployment.

References

[1] E. Bonabeau, M. Dorigo, G. Theraulaz, Swarm Intelligence:From Natural to Artificial Systems, Oxford University Press, Ox-ford, 1999.

[2] G. Conzelmann, V.S. Koritarov, K. Guziel, T.D. Veselka, FinalAudit Report—New Generating Capacity Tenders 97/1 and 97/2,Argonne National Laboratory, 1999.

[3] S. Gallopoulos, E. Houstis, J. Rice, Computer as thinker/doer:problem-solving environments for computational science, IEEEComput. Sci. Eng. 1 (2) (1994) 11–23.

[4] Harza Engineering Company, Trans-Balkan Power Line Project,Final Report, Argonne National Laboratory, 2001.

[5] V. Koritarov, G. Conzelmann, T. Veselka, W. Buehring, R.Cirillo, Incorporating environmental concerns into electric sys-tem expansion planning using a multi-criteria decision supportsystem, Int. J. Glob. Energy Issues 12 (1–6) (1999) 60–67.

[6] M. North, V. Koritarov, G. Boyd, T.D. Veselka, C.M. Macal, G.C.Conzelmann, P.R. Thimmapuram, E-Laboratories: agent-basedmodeling of electricity markets, in: Proceedings of the AmericanPower Conference, PennWell, 2002.

[7] J.C. VanKuiken, T.D. Veselka, K.A. Guziel, D.W. Blodgett,S. Hamilton, J.A. Kavicky, V.S. Koritarov, M.J. North, A.A.Novickas, K.R. Paprockas, E.C. Portante, D.L. Willing, Argonne

Sys-

[ J.C..A.as,etingbo-

es-

atLgorgy.ce

im-esin-

t oret ional

Domain objects include both those responsibleunctional behavior in the PSE and those objectsre the subject of the analysis—usually data obf some kind. The user can therefore watch a disf the PSE elements evolving during use, improv

heir understanding of the analysis process, e.g. wolver, lookup table, data set, etc., is being used. Aame time, the change in state of each actual agese and the values that the agent represents can aisplayed.

. Conclusion

ABMs allow the use of complex adaptive systepproaches that are capable of producing tools or

hat can address the behavior of each of the participithin complex systems. In this paper, we argued

Production, Expansion, and Exchange Model for Electricaltems User’s Guide, Argonne National Laboratory, 1994.

8] T.D. Veselka, E.C. Portante, V.S. Koritarov, S. Hamilton,VanKuiken, K.R. Paprockas, M.J. North, J.A. Kavicky, KGuziel, L.A. Poch, S. Folga, M.M. Tompkins, A.A. NovickImpacts of western area power administration’s power markalternatives on electric utility systems, Argonne National Laratory, 1994.

M. North is the Deputy Director of thCenter for Complex Adaptive Agent Sytem Simulation (CAS2) within the Decisionand Information Sciences Division (DIS)Argonne National Laboratory (ANL). ANis operated by the University of Chicafor the United States Department of EneMr. North has over 13 years of experiendeveloping advanced modeling and sulation applications for various branchof the federal government and several

ernational agencies. In addition, Mr. North has published mhan 20 papers on ABMS in refereed journals and internat

1198 M. North et al. / Future Generation Computer Systems 21 (2005) 1192–1198

conferences. Mr. North is a graduate of the Illinois Institute ofTechnology. Mr. North holds degrees in a variety of subjects in-cluding Computer Science, Computer Systems Engineering andMathematics.

C. Macal is the Director of CAS2. Dr.Macal is involved with developing new ap-plications and projects in complex adap-tive systems and agent-based simulation toproblems of national interest, in such fieldsas business processes and supply chains,national infrastructure and electric powermarkets, the hydrogen infrastructure, bio-logical systems and chemotaxis networks,social systems and terrorism. Dr. Macal re-ceived a PhD in Industrial Engineering and

Management Sciences from Northwestern University and holds anMS in Industrial Engineering and a BS in Engineering Sciences from

Purdue University. He is a registered Professional Engineer in thestate of Illinois and a Member of the Society for Computer Simula-tion International (SCSI), the Institute for Operations Research andthe Management Sciences (INFORMS), and the Systems DynamicsSociety.

P. Campbell was formerly the Directorof ANL’s DIS Division. Peter has been insemi-retired since 2000 and he works forANL as a part-time consultant. Peter wasborn and educated in Australia, where helectured at Newcastle and Macquarie Uni-versities in New South Wales. Peter haspreviously worked for US engineering con-sulting firm Dames and Moore, and joinedANL in 1986. For the past 8 years, Camp-bell’s group at ANL has been working onthe use of computers to simulate humanbehavior.