cooperative agents networksjmas.us.es/doc/2011/03.Vinyals-JMAS2011.pdf · University of Verona...

Local optimization in cooperative agents networks

Dott.ssa Meritxell VinyalsUniversity of Verona

Seville, 30 de Junio de 2011jueves 28 de julio de 2011

Outline

. Introduction and domains of interest

. Open problems and approaches

jueves 28 de julio de 2011

Outline

Many real-world problems can be modeled as a network of cooperative agents that have to coordinate their actions in order to

optimize the system performance

Traffic light control

Energy-efficientsensor networksSmart (electricity) grid

Motivating domains

Motivating domainsIntroductionApplications

Open problems

SchedulingSmart gridTraffic light synchronization

DCOPs for smart grid: Configuration of power networks

[Petcu & Faltings, 2008]

How sinks configure the network by enabling transmission lines suchthat is (a) cycle-free and (b) the amount of power lost is minimized.

Meritxell Vinyals Multi-Agents systems for Distributed Optimization

IntroductionApplications

Open problems

Motivating domains

Optimization with agents

Each agent can choose from a set of discrete actions

Each agent sensor has two actions:

Turn on Turn off

Each agent sensor has two actions:jueves 28 de julio de 2011

Agents report individual rewards for their actions

1/2$ 1/2$

Agent sensors report a reward to turn off (e.g. which may vary depending on the remaining battery)

An edge stands for two agents that need to coordinate in order to receive a joint reward

There is a reward if the region between two sensors is sampled by at least one sensor

If two neighboring agents select different actions they receive 1$

The goal is to distributedly find a set of actions that maximize the overall reward

• Optimal configuration 3$

1/2$ 1/2$

Disaster management Traffic light control

Motivating domains

Coordinate traffic lights so that vehicles can traverse an arterial in one traffic direction, keeping a specified speed without stopping (green waves)

Old times: isolated traffic lights

Future generation:social traffic lights

Traffic light control [R. Junges and A. L. C. Bazzan, 2010] uses a multi-agent system approach in which:22 Chapter 2. Problem definition

1st. st.

2nd. st.

(a) Traffic light synchronization

problem.

r13 � ⊕� 0 1

⊕ 32 2

r02 � ⊕� 1

⊕ 1 2

r01 � ⊕� 0

⊕ 132

r23 � ⊕� 0

⊕ 132

(b) DCOP model of the traffic light synchronization

problem.

Figure 2.4: An traffic light synchronization problem. (a) The traffic light synchroniza-

tion problem: there are four crossings with an square connection. (b) The DCOP model:

each crossing has an associated variable that models the feasible plans for the crossing.

sensors s1 and s2 in order to avoid redundant measurements and minimise energy con-

sumption. Figure 2.3(b) shows a DCOP model of the problem. The variables represent

sensors and each variable’s domain is composed of sensor’s actions (to turn on or off).

Unary relations stand for sensors costs, e.g. unary relation r0 encodes a reward of12

for sensor x0 to turn on. Binary relations stand for sensors dependencies, relation r01

encodes the rewards for the joint configurations of sensors s0 and s1.

As argued in chapter 1 for the specific case of environmental monitoring, this kind

of sensor networks are typically composed of a large number of small-sized battery-

operated sensors equipped with limited data processing and communication capabilities

and hence, coordination to the optimal is in general not affordable under such severe

conditions. Moreover, information gathering is a domain that allows to select among

different on sensor positions or formations to trade-off quality versus cost.

2.3.3 Traffic light controlConsider the problem of synchronizing traffic lights in a city. Synchronization here

means that adjacent traffic lights will be coordinated so that vehicles can traverse an

arterial in one traffic direction, keeping a specific speed, without stopping. Each traffic

light has a library of plans, each allowing the synchronization in different traffic direc-

tion. The specific objective of the scenario is to find the best signal plan configuration

for the traffic lights.

This problem of traffic light synchronisation is formalized in (Junges and Bazzan,

2008) as a DCOP Φ = �A,X ,D,R� such that:

• X = {x1, . . . , xn} and A = {a1, . . . , an} are the sets of variables/agents, where

n is the number of crossings;

1st. st.

2nd. st.

problem.

r13 � ⊕� 0 1

⊕ 32 2

r02 � ⊕� 1

⊕ 1 2

r01 � ⊕� 0

⊕ 132

r23 � ⊕� 0

⊕ 132

problem.

Each agent is in charge of a crossing

1st. st.

2nd. st.

problem.

r13 � ⊕� 0 1

⊕ 32 2

r02 � ⊕� 1

⊕ 1 2

r01 � ⊕� 0

⊕ 132

r23 � ⊕� 0

⊕ 132

problem.

The decision of an agent is composed of a set of signals plans for the traffic lights in

the crossing

⊗⊕

Traffic light control [R. Junges and A. L. C. Bazzan, 2010] uses a multi-agent system approach in which:

Two agents in two adjacent crossings need to coordinate

their plans

The reward to execute two plans in neighboring crossings is in

function of:(1) the degree in which these two

plans synchronize(II) the volume of vehicles in that

direction

⊗⊕

⊕2$ 1$2$ 0$

Smart grid. The current hierarchical, centrally-controlled grid is obsolete. Problems on scalability, efficiency and integration of green energies. Most of the decisions about the operation of a power system are made in a centralized fashion

Smart grid. Centralized control is replaced with decentralized control:

. efficiency and scalability . complex control mechanisms needed

. Introduction of intelligence at all levels, especially at lower levels, to provide timely and accurate control responses

Smart grid

. Home/neighborhood level

. Distribution level

. Transmission level

Smart grid

. Home/neighborhood level

. Coordinate home appliances to reduce the peak

. Create coalitions of energy profiles to reduce the peak

Smart grid. Transmission level

How sinks configure the network by enabling transmission lines such that are: (a) cycle free; and (b) the amount of power lost is minimized.

Open problems

Outline

Maxims for researchersFirst takes all the credit, second gets nothing

Maxims for researchersEither you are the first or you are the best in the crowd

The moral

Identify open problems, preferably with few contributions

Open problems

Trade-off quality vs cost

Dynamism

Uncertainty

Hierarchical optimization

Non-cooperativeagents

Open problems

Trade-off quality vs cost

Dynamism

Uncertainty

Hierarchical optimization

Non-cooperativeagents

Optimality: the idealistic (but usually impractical) term

Researchers have proposed optimal algorithms that aim to minimize the communication/computation needed by agents to find their optimal actions

All of them have an exponential cost (either in size/number of messages/computation)

DPOP [A. Petcu & B. Faltings, 2005] ADOPT [Modi et al., 2005] OptAPO [R. Mailler & V. Lesser, 2004] Action-GDL [M. Vinyals et al., 2010]

Smart grid Traffic light control Energy-efficientsensor networks

Time Scale Communication/Computation

In many domains the price of optimality is simply not affordable

Suboptimality: low-cost at not guarantees

Researchers have also proposed suboptimal algorithms:

. Return fast good solutions in average . Small amount of communication/computation per agent

But not guarantee ....

DSA [Yokoo & Hirayama, 1996] DBA [Fitzpatrick & Meeterns., 2005] Max-Sum [Farinelli et al., 2009]

Although suboptimal coordination returns good solutions on average ....

2.75$(Optimal configuration 3$)

... it can also converge to very poor solutions

1.25$(Optimal configuration 3$)

1/2$1/2$

Suboptimal coordination with quality guarantees

A quality guarantee ensures that the value of a solution is within a given distance δ from the optimal one

δ ≤ solution valueoptimal value

δ= 50%

1/2$1/2$

Quality guarantees allow system designer to evaluate different design alternatives: algorithm selection

e.g. in traffic control Algorithm A Algorithm B

The best solution varies with traffic conditions ....

... but the structure of dependencies is fixed and

determined by the particular urban grid

e.g. in traffic control

Algorithm A Algorithm B50% 25%

e.g. in traffic control

Algorithm A Algorithm B20% 40%

Quality guarantees allow system designer to evaluate different design alternatives: configuration selection

e.g. in sensor networks

We can select a placement for

sensors

Quality guarantees allow system designer to evaluate different design alternatives: configuration selection

e.g. in sensor networks

We can select a placement for

sensors

Suboptimal coordination with quality guarantees: approaches

• Region optimal algorithms [AAMAS, 2011]

A solution is region optimal when its value cannot be improved by changing the decision of any group of agents in the region

{a0, a1}

Groups in the region

{a0, a2}{a0, a3}

{a1, a2}{a1, a3}

{a2, a3}

{a1, a2, a3}{a0, a3}

{a0, a1, a2}

Groups in the region

Region optimality [AAMAS, 2011] allows to assess quality guarantees for any region optimal

The quality of any region optimal in this region in any

problem is guaranteed to be at least 33% the value of the

optimal solution

problem with this dependency graph is guaranteed to be at least 50% the value of the

optimal solution

problem with this dependency graph and reward structure is guaranteed to be at least 75%

the value of the optimal solution

So far we have characterized quality guarantees for region optimal solutions but ...

how agents find such region optimal solutions?

So far we have characterized quality guarantees for region optimal solutions but ...

how agents find such region optimal solutions?

A generic region optimal algorithm

Initialization Optimization Implementation

Agents select an initial action

1/2$ 1/2$

Each group of agents in the region optimizes its decision given other agents decisions.

1/2$ 1/2$

The quality of any region optimal in this region in any problem is guaranteed to be at least 50% the value of the

optimal solution

... until stabilitzation

Message to take away

• Many real-world problems can be modeled as a network of agents that need to coordinate their actions to optimize system performance

• Optimality is not affordable in many of these emerging large-scale domains

• An open line of research is how to design suboptimal algorithms that provide quality guarantees over the agent’s actions

Gracias por vuestra atención!!!

cooperative agents networksjmas.us.es/doc/2011/03.Vinyals-JMAS2011.pdf · University of Verona...

Documents

PROYECTOS Y EQUIPOS COMPLETOS PARA EL TRATAMIENTO DEL AGUA Carretera Mollet-Sabadell, Km. 4,3 - Polígono Can Vinyals, Nave 6 - 08130 SANTA PERPÈTUA DE

¡logo-Exo-2014_2015.… · GRUPO GESTIÓN INTEGRAL NOVOLUX INTERNACIONAL, SL. C/ Ramón Berenguer, 8 (Polígono Industrial Can Vinyals) 08130 Sta. Perpetua de Mogoda - BARCELONA

WORKSHOP PROGRAMC4.6, Video Games and Machine Learning Synnaeve, Togelius, Schaul, Vinyals, Usunier C4.7, ML on a budget: IoT, Mobile and other tiny-ML applications Varma, Saligrama,

Abstract arXiv:1608.05343v1 [cs.LG] 18 Aug 2016 · Decoupled Neural Interfaces using Synthetic Gradients Max Jaderberg Wojciech Marian Czarnecki Simon Osindero Oriol Vinyals Alex

Matching Networks for One Shot Learning - arXiv · Matching Networks for One Shot Learning Oriol Vinyals Google DeepMind vinyals@google.com Charles Blundell Google DeepMind cblundell@google.com

Classiﬁcation Accuracy Score for Conditional …Classiﬁcation Accuracy Score for Conditional Generative Models Suman Ravuri, & Oriol Vinyals DeepMind London, UK N1C 4AG ravuris,

Video Pixel Networks - arXiv.org e-Print archive Pixel Networks Nal Kalchbrenner, A aron van den Oord, Karen Simonyan Ivo Danihelka, Oriol Vinyals, Alex Graves, Koray Kavukcuoglu fnalk,avdnoord,simonyan,danihelka,vinyals,gravesa,koraykg@google.com

Formación Arquitectos Restauradores. Modelo Pedagógico]PROYECTO DE TESIS LA FORMACIÓN DE LOS ARQUITECTOS RESTAURADORES. BÚSQUEDA DE UN MODELO PEDAGÓGICO. Mariona Genís Vinyals

Abstract arXiv:2007.03629v2 [cs.LG] 8 Jul 2020Strong Generalization and Efﬁciency in Neural Programs Yujia Li, Felix Gimeno, Pushmeet Kohli, Oriol Vinyals DeepMind {yujiali,fgimeno,pushmeet,vinyals}@google.com

1 Deep Audio-Visual Speech Recognition · 1 Deep Audio-Visual Speech Recognition Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, Andrew Zisserman Abstract—The

Inteligencia sociemocion alen MASjmas.us.es/doc/2011/02.Romero-Ternero-JMAS2011.pdf · 2011-08-01 · Mª del Carmen Romero Ternero mcromerot@us.es ... procesamiento de lenguaje natural

Video Pixel Networks - arXivVideo Pixel Networks Nal Kalchbrenner, A aron van den Oord, Karen Simonyan Ivo Danihelka, Oriol Vinyals, Alex Graves, Koray Kavukcuoglu fnalk,avdnoord,simonyan,danihelka,vinyals,gravesa,koraykg@google.com

Deep Chit-Chat: Deep Learning for ChatBots · Representations using RNN Encoder-Decoder for Statistical Machine Translation. EMNLP’. •Ilya Sutskever, Oriol Vinyals, and Quoc V

Towards a Realistic Bid Generator for Mixed Multi …pst.istc.cnr.it/RCRA07/articoli/P23-vinyals-etal-RCRA07.pdfTowards a Realistic Bid Generator for Mixed Multi-Unit Combinatorial

ALPARGATAS BASES PIZZA BOCADILLOS CROQUE … · Industrias de Bollería y Pastelería LACAMBRA S.A. Polígono Molí Vinyals s/n 08776 Sant Pere de Riudebitlles (Barcelona) TEL/FAX

Deep Learning - Sciencesconf.orgcbdcom2016.sciencesconf.org/conference/cbdcom2016/... · Fig from Vinyals et al, Google April 2015 NIC Generator Fig from Graves, Schmidhuber et al,

Bird Species Categorization Using Pose Normalized Deep ...vision.cse.psu.edu/people/chrisF/deep-learning/... · arXiv:1406.2952v1 [cs.CV] 11 Jun 2014 {Donahue, Jia, Vinyals, Hoffman,

Model vs Optimization Meta Learningmetalearning-symposium.ml/files/vinyals.pdf · Oriol Vinyals, NIPS 17 Definition of Meta Learning What is Meta Learning / Learning to Learn? Go

CSC412: Variational Autoencodersduvenaud/courses/csc412/...• Generating Sentences from a Continuous Space. Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz,

Agentes y Sistemas Multi-Agente en la Empresa Actualjmas.us.es/doc/2011/10.Pinyol-JMAS2011.pdf• Utillajes de fabricación(moldes, matrices). • Microtecnologías. • Robótica