Slide 1

Real-Time Neuroevolution in the NERO Video GameMike Taks Bram van de KlundertAboutPublished 2005Cited 286 timesKenneth O. StanleyAssociate Professor at University of Central FloridaRisto MiikkulainenProfessor at the University of Texas at AustinBobby D. BryantAssistant Professor at university of NevadaKenneth O. Stanley created NEAT, also recomened NERO project2ContentsIntroductionNEATrtNEATNEROIntroductionreal-time NeuroEvolution of Augmenting Topologies (rtNEAT)Adaption of the NEAT algorithmCreate new genre of games requiring learningBlack and whiteTamagotchi

NEATNeuro Evolution of Augmenting TopologiesGrowing neural network

RepresentationList of connection genesInnovation numberGlobal counterIn nodeOut nodeWeightEnabled

Initial populationUniform population of simple networksNo hidden nodesRandom weightsMutationWeight mutationStructure mutationAdd a connection between two nodesReplace a connection by a nodeConnection not removed only disabledOut connection inherits the valueCrossover termsDisjoint: gene is only in one networkExcess: disjoint and outside of the range of innovations

Crossovershared genes:Uniform crossoverBlend crossoverDisjoint and excess genesTaken from most fit parentCrossover exampleEqual fitness9, 10 excess6, 7 , 8 disjoint

SpeciationSpeciationSpecies assignmentCheck if there is a species close enough to the individualIf not, create new species FitnessSelectionTrailer NEROrtNEATDifferencesSelection and replacementRemoving agentDifferencesWork real timeOriginally NEAT evaluates one complete generation of individuals, generates offspring en masseDifferencesDuring a game, performance statistics are being recordedReplacing agentsPerform actions every n game-ticks

Selection and replacementCalculate fitnessRemove worst agent of sufficient ageChoose parents among the bestCreate offspring, Reassign all agents to species

Removing agentRemove worst agent based fitness adjusted for species sizeNew agents are continuously born, life time individually kept track ofPossibility to just replace the neural network of an agentNEROPlayer is a trainerSet up exercisesSave and load neural networks

Training modePlace objects on the field (static enemies, turrets, rovers, flags, ...)Adjust fitness rewards by sliders

Training modeAgents spawn in the factory

SensorsRadar to track enemy locationRangefinderLine-of-fire...

Evolved topology

Battle modeAssemble team of 20 agentsEnds if one team is empty

ExperimentsSlightly nondeterministic game engineThe same game is thus never played twice

BehaviorsDifferent behaviors are trainedSeek and fire by placing a single static enemy on the training fieldFiring and hitting a target was to slow to evolve. Aiming script was usedBehaviorsAvoidance trained by controlling an agent manuallyAgent runs backwards facing the enemy and shooting at it

BehaviorsTrain agents to avoid turret fire

More complex behaviorsLet agents attack enemy behind a wallTrain agents to avoid hazardous corridors

More complex behaviorsTrain agents versus targets that are standing against a wall

More complex behaviorsIncrementally add walls, agents will be able to navigate

BattlingPaper rock scissorsSeek vs avoidance

Battling36Questions?