Artistic Robots through Interactive Genetic Algorithm with ELO rating system Andy Goetz, Camille Huffman, Kevin Riedl, Mathias Sunardi and Marek Perkowski

Artistic Robots through Interactive Genetic Algorithm

with ELO rating system

Andy Goetz, Camille Huffman, Kevin Riedl, Mathias Sunardi and Marek

Perkowski

Department of Electrical Engineering, Portland State University

Portland Portland Cyber Cyber

TheatreTheatre

Making Making science out science out

of robot of robot theater?theater?

How to make a science from How to make a science from robot theatre?robot theatre?

We want to evaluate sound, shape, motion, color,etc.

Behavior Generation and Verification

Interactive Genetic

Algorithm

Behavior Behavior expressioexpressio

nn

Behavior Behavior AutomatonAutomaton

Probabilistic Automaton

behavior Generator and verifier

robotrobot

Human Human evaluatorsevaluators

Main Idea of this Main Idea of this paper paper

• A new approach to create fitness function for Interactive Genetic Algorithm in which (possibly) many humans evaluate robot motions via Internet page.

• Based on ELO rating system known from chess.

• The robots use:1. a genetic algorithm, 2. fuzzy logic, 3. probabilistic state machines, 4. a small set of functions for creating picture

components, 5. and a user interface which allows the Internet users to

rate individual sequences.

Previous work on IEC systems

1. Human-based genetic algorithm.

2. Interactive evolution strategy,

3. Interactive genetic programming,

4. Interactive genetic algorithm.

Mostly for music composition and graphics

Usually weighted functions were used

Ranking Systems in Sports

Rating systems for many sports award points in accordance with subjective evaluations of the 'greatness' of certain achievements.

For example, winning an important golf tournament an important golf tournament might be worth an arbitrarily chosen five times as many points as winning a lesser tournament.

A statisticalstatistical endeavor, by contrast, uses a model that relates the game results to underlying variables representing the ability of each player.

Elo rating systemElo rating system

• The Elo rating system is a method for calculating the relative skill levels of players in two-player games such as chess.

• It is named after its creator Arpad Elo, a Hungarian-born American physics American physics professor.

• The Elo system was invented as an improved chess rating system, but today it is also used in many other games.

• It is also used as a rating system for multiplayer competition in a number of video games.

• It has been adapted to team sports including association football, American college football, basketball, and Major League Baseball.

http://www.answers.com/topic/chess-rating-systems

http://www.answers.com/topic/video-game

http://www.answers.com/topic/video-game

http://www.answers.com/topic/soccer

http://www.answers.com/topic/major-league-baseball-1

Previous Previous worksworks

Pairwise Pairwise ComparisoCompariso

nn

Pairwise Comparison

Method:Method:

Compare each two candidates (players) head-to-head.

Award each candidate one point for each head-to-head victory.

The candidate with the most points wins.

N(N-1)/2 comparisons.

Pairwise Comparison - Example

Selection of best robot best robot facial expressionfacial expression:

4 candidates: {A,B,C,D} and 4 rankings of them

37 voters

5 trials (columns)

Table shows the rankings of the candidates (rows) and the number of voters (columns) that ranked the candidates that way

# of Voters

Rank 14 10 8 4 1

1st A C D B C

2nd B B C D D

3rd C D B C B

4th D A A A A


Compare candidates A & B:

14 voters ranked A higher than B

10+8+4+1=23 voters ranked B higher than A

So, B wins against A

# of Voters

Rank 14 10 8 4 1

1st A C D B C

2nd B B C D D

3rd C D B C B

4th D A A A A


Next, compare candidates A & C:

14 voters ranked A higher than C

10+8+4+1=23 voters ranked C higher than A

So, C wins against A

Continue for next pairs: A vs. D, B vs. C, B vs. D, C vs. D

Exclude:

permutations (e.g. C vs. A = A vs. C)

comparison with itself (e.g. A vs. A)

# of Voters

Rank 14 10 8 4 1

1st A C D B C

2nd B B C D D

3rd C D B C B

4th D A A A A


Record points:

wins=1, lose=0

# of Voters (total=37)

Rank 14 10 8 4 1

1st A C D B C

2nd B B C D D

3rd C D B C B

4th D A A A A

A B C D Wins over

Lost agains

tPoints

A 14 14 14

B 23 18 28

C 23 19 25

D 23 9 12

Cell values: number of voters Cell values: number of voters that ranked candidate (row) that ranked candidate (row)

over candidate (column)over candidate (column)


Record points:

wins=1, lose=0


Rank 14 10 8 4 1

1st A C D B C

2nd B B C D D

3rd C D B C B

4th D A A A A

A B C D Wins over

Lost agains

tPoints

A 14 14 14 - B,C,D 0

B 23 18 28

C 23 19 25

D 23 9 12




Record points:

wins=1, lose=0


Rank 14 10 8 4 1

1st A C D B C

2nd B B C D D

3rd C D B C B

4th D A A A A

A B C D Wins over

Lost agains

tPoints

A 14 14 14 - B,C,D 0

B 23 18 28 A,D C 2

C 23 19 25 A,B,D - 3

D 23 9 12 A B,C 1




Record points:

wins=1, lose=0


Rank 14 10 8 4 1

1st A C D B C

2nd B B C D D

3rd C D B C B

4th D A A A A

A B C D Wins over

Lost agains

tPoints

A 14 14 14 - B,C,D 0

B 23 18 28 A,D C 2

C 23 19 25 A,B,D - 3

D 23 9 12 A B,C 1



C wins!C wins!


Record points:

wins=1, lose=0


Rank 14 10 8 4 1

1st A C D B C

2nd B B C D D

3rd C D B C B

4th D A A A A

A B C D Wins over

Lost agains

tPoints

A - B C D - B,C,D 0

B - - C B A,D C 2

C - - - C A,B,D - 3

D - - - - A B,C 1

Another way to calculate the winner: use Another way to calculate the winner: use half the table triangle, mark the winner, half the table triangle, mark the winner,

and count the number of times the player and count the number of times the player appearsappears

C wins!C wins!

Other possible scenario

A three-way tie:

Inconsistency:

A wins over B, B wins over C, C wins over A

A B C

A - A C

B - - B

C - - -

ELO ELO Rating Rating SystemSystem

Overview of ELO

A player’s skill is assumed to be a normal distribution:

True skill is around the mean

Elo System gives two things:

A players expected chance of winning

A method to update a player’s Elo Rating

Basic Ideas of ELO• One cannot look at a sequencecannot look at a sequence of moves and say, "That

performance is 2039."

• Performance can only be inferred from wins, draws and losses.

• Therefore, if a player wins a game, he is assumed to have performed at a higher level than his opponent for that game.

• Conversely if he loses, he is assumed to have performed at a lower level.

• If the game is a draw, the two players are assumed to have performed at nearly the same level.

A player’s ranking is updated based on its:

Expected value of winning (E)

Which depends on the ranking difference with the opponent

Outcome of the match (S for ‘score’)

1 = win

0 = lose

0.5 = draw

Scores and ranking of Scores and ranking of playersplayers

Expected score (E)

Where:

EA, EB = expected score for player A and B, respectively

RA, RB = Rating of player A and B, respectively

Remember: 1=win, 0=lose, 0.5=draw

Expected scores in Elo Rating

http://en.chessbase.com/home/TabId/211/PostId/4007114

scorescore

ratingrating

http://en.chessbase.com/home/TabId/211/PostId/4007114

Characteristics of ELOA player with higher Elo ranking than his opponent has a higher expected value (i.e. chance of winning), and vice versa.

When both players have similar Elo rankings, the chance of having a draw having a draw is higher.

After the match, both players’ rankings are updated with the same amount, but:

the winner gains the rank (rating),

the loser loses the rank.

If a higher ranking player (‘stronger’) wins against a weaker player, the rank changes are smaller than when the weaker player wins against the higher ranking player.

Subjective value K

Basic Assumptions of ELOElo's central assumption was that the chess performance of each player in each game is a normally distributed normally distributed random variable. random variable.

Although a player might perform significantly better or worse from one game to the next, ELO assumed that the mean value of the performances of any given player changes only slowly over time.

A further assumption is necessary, because chess performance in the above sense is still not measurable.

Our question: “Is ELO good for human evaluation of robot art (motion, behavior)?”

How ELO WorksHow ELO WorksA player's expected score is his probability of probability of winning winning plus halfhalf his probability of drawing.

Thus an expected score of 0.75 could represent a 75% chance of winning, 25% chance of losing, and 0% chance of drawing.

On the other extreme it could represent a 50% chance of winning, 0% chance of losing, and 50% chance of drawing.

The probability of drawingprobability of drawing, as opposed to having a decisive result, is not specified in the Elo system.

Instead a draw is considered half a win and half a loss.

How ELO Works

The relative difference in rating between two players determines an estimate for the expected score between them.

Both the average and the spread of ratings can be arbitrarily chosen.

Elo suggested scaling ratings so that a difference of 200 rating points in chess would mean that the stronger player has an expected score (which basically is an expected average score) of approximately 0.75,

The USCF initially aimed for an average club player to have a rating of 1500.

Elo Rating - Elo Rating - ExampleExample

Suppose a Robot Boxing league:

The league has tens, hundreds, or more robots

Each robot has a ranking (higher number = higher rank)

A robot’s ranking is updated after each match

But it can also be done after multiple matches

A match is a one-vs-one battle

Expected score (E)

Suppose:

Robot A rank: 1500

Robot B rank: 1320

Then:

EA = 1 / (1 + 10(1320 - 1500)/400) = 0.738

EB = 1 - 0.738 = 0.262

Elo Rating Example: scores for robots

Expected Expected to winto win

Elo Rating Example: Adjusting ratings after

matchNext, the match is held.

After the match, the ratings of both robots will be adjusted by:

Where:

R’A = Robot A’s new rating

RA = Robot A’s old/current rating

K = some constant*, for practical reasons we choose K=24 in this example

S = Score/match result (1=win, 0=lose, 0.5=draw)

EA = Expected score

Similarly for robot B

Elo Rating Example: Adjusting scores after one

matchSuppose the outcome of the match:

Robot A wins!

Robot B wins!

It’s a draw!

Robot A rank: 1500

Robot B rank: 1320

Remember before the match it was:

Elo Rating Example: adjusting rankings after five

matchesSuppose rank update is done after 5 matches:

Robot A current rank: 1500

Opponent/matchOpponent rank

(RB)EA

Score/match outcome (1=win, 0=lose,

0.5=draw)

1 1320 0.738 1

2 1700 0.240 1

3 1480 0.529 0

4 1560 0.415 0.5

5 1800 0.151 0

Total 2.073 2.5

About K in chessK is the rate of adjustments to one’s rating.

Example when Robot A wins (B loses):

Some Elo implementations adjust K based on some criteria. For example:

FIDE (World Chess Federation):

K = 30 for a player new to the rating list until s/he has completed events with a total of at least 30 games.

K = 15 as long as a player's rating remains under 2400.

K = 10 once a player's published rating has reached 2400, and s/he has also completed events with a total of at least 30 games. Thereafter it remains permanently at 10.

USCF (United States Chess Federation):USCF (United States Chess Federation):

Players below 2100 --> K-factor of 32 used

Players between 2100 and 2400 --> K-factor of 24 used

Players above 2400 --> K-factor of 16 used.

How about robot art?

Picture Picture Drawing Drawing RobotsRobots

Audience votes through a Webpage

ELO for art (motion) scoring

Score of 194

Score of 0

ELO for art (motion) scoring

Physical Robot DERPY

Derpy with a sharpie marker

Fuzzy/Probabilistic state Machine operates differently in dark and light

areas.

Image with dark and light areas.

Examples of fuzzy variables.

Fuzzy and Fuzzy and Probabilistic Probabilistic

MachinesMachines

Simple probabilistic machine of Derpy

“Robot art” on butcher paper located on a floor.

Another piece of art from Derpy

Now use Part 2 of slides

Auxiliary Auxiliary SlidesSlides

MicrosofMicrosoft t

TrueSkillTrueSkill

Microsoft TrueSkillAddressing:

Subjective K value - instead, based on players’ skill

Ranking of multiple players (>2)

Can find “interesting” matches - balanced, where either player have comparable chance of winning the match.

Build “Leaderboards” (ranking of all players)

Microsoft TrueSkillPlayer’s skill is modeled as normal distribution, with mean as the player’s “true skill” and standard deviation as uncertainties (about the player’s skill)

Player start with some “mean skill” and uncertainty values.

As player plays more games/matches, the mean skill gets adjusted, and the uncertainty (i.e. std. dev) decreases.

Microsoft TrueSkill

Updating mean and standard deviation

ββ22 is unknown, which is unknown, which is the variance of the is the variance of the performance around performance around

the skill of each the skill of each player.player.

Microsoft TrueSkill

v and w

Microsoft TrueSkill

Microsoft TrueSkill

Microsoft TrueSkill http://research.microsoft.com/en-us/projects/trueskill/Details.aspx#How_to_Update_Skills

unbalanced matches (can't win or can't lose) are not interesting

balanced matches are interesting (even chance of winning)

accommodates two or more players

a module to track skills of all players based on game outcomes between players (update)

TA module to arrange interesting matches for its members (Matchmaking)

module to recognize and potentially publish skills of members (leader boards)

Truskill is skill-based ranking system

so interesting matches can reliably arranged within a league

uses Bayesian inference for ranking

http://research.microsoft.com/en-us/projects/trueskill/Details.aspx#How_to_Update_Skills

http://research.microsoft.com/en-us/projects/trueskill/Details.aspx#How_to_Update_Skills

Microsoft TrueSkill• The intuition is that the greater the difference between two player’s μ values

– assuming their σ value are similar – the greater the chance of the player with the higher μ value performing better in a game.

• This principle holds true in the TrueSkill ranking system. But, this does not mean that the players with the larger μ's are always expected to win, but rather that their chance of winning is higher than that of the players with the smaller μ's.

• The TrueSkill ranking system assumes that the performance in a single match is varying around the skill of the player, and that the game outcome (relative ranking of all players participating in a game) is determined by their performance.

• Thus, the skill of a player in the TrueSkill ranking system can be thought of as the average performance of the player over a large number of games.

• The variation of the performance around the skill is, in principle, a configurable parameter of the TrueSkill ranking system.

Microsoft TrueSkill

mu and sigma are updated based on outcome of game (win/lose). score difference makes no impact.

1. assumes skill of each player may change slightly between current and previous game -> sigma is increased (a configurable parameter) "It is this parameter that both allows the TrueSkill system to track skill improvements of gamers over time and ensures that the skill uncertainty σ never decreases to zero ("maintaining momentum")."

2. determine the probability of game outcome for given skills of participating players, and weight by probability of corresponding skill beliefs. --> average over all possible performances (weighted by their probability - Bayes Law) and derive the game outcome from performances: player with highest performance is winner, second highest is first tuner up, and s on.

3. if player performance are very close, true skill considers the outcome to be draw. The larger the draw margin is defined in a league, the more likely a draw is to occur. The size of margin is configurable and adjusted by game mode.

Measuring Measuring consistencconsistenc

yy

Measuring consistency in

Pairwise ComparisonCan be done when comparison is done with “degree of importance”. E.g.:

1=equally important, 2=somewhat more important, 3=more important, 4=most important

Example:

Determining important criteria in buying a car

Price MPG Comfort Style

Price 3 2 2

MPG

Comfort 4

Style 4 2Values in cells are Values in cells are importance with importance with

respect to the row respect to the row itemitem


Pairwise ComparisonComplete the values in the matrix

Example:

Determining important criteria in buying a carPrice MPG Comfort Style

Price 1 3 2 2

MPG 1

Comfort 4 1

Style 4 2 1Criterion compared Criterion compared to itself is “equally to itself is “equally

important”important”



Pairwise ComparisonComplete the values in the matrix

Example:

Determining important criteria in buying a carPrice MPG Comfort Style

Price 1 3 2 2

MPG 1/3 1 1/4 1/4

Comfort 1/2 4 1 1/2

Style 1/2 4 2 1


Importance of less important criterion is reciprocal of the importance of the more important Importance of less important criterion is reciprocal of the importance of the more important criterioncriterion

e.g.: Price vs. Style => Style is two times more imporant than Price (2). So, Price is one half e.g.: Price vs. Style => Style is two times more imporant than Price (2). So, Price is one half as imporant than Style (1/2)as imporant than Style (1/2)


Pairwise ComparisonExample:

Determining important criteria in buying a car

Calculate weights of each criterion:

Price MPG Comfort Style

Price 1 3 2 2

MPG 1/3 1 1/4 1/4

Comfort 1/2 4 1 1/2

Style 1/2 4 2 1

Evaluation Criteria for Ranking Methods

The Method of Pairwise Comparisons satisfies the Majority Criterion. (A majority candidate will win every pairwise comparison.)

The Method of Pairwise Comparisons satisfies the Condorcet Criterion. (A Condorcet candidate will win every pairwise comparison -- that's what a Condorcet candidate is!)

The Method of Pairwise Comparisons satisfies the Public-Enemy Criterion. (If there is a public enemy, s/he will lose every pairwise comparison.)

The Method of Pairwise Comparisons satisfies the Monotonicity Criterion. (Ranking Candidate X higher can only help X in pairwise comparisons.)

ELOELO

Agenda on ELO

Overview

How it works

Details

Mathematical details

How it Works

Elo formulas

Expected value

Score

How to update the rank

Documents

Artistic Robots through Interactive Genetic Algorithm with ELO rating system Andy Goetz, Camille Huffman, Kevin Riedl, Mathias Sunardi and Marek Perkowski