Modeling Space Fortress Abraham Anderson John Anderson Shawn Betts Dan Bothell Jennifer Ferris Jon...

Modeling Space FortressAbraham Anderson

John AndersonShawn BettsDan Bothell

Jennifer FerrisJon Fincham

Michelle MoonBen PooleYulin Qin

The Decomposition Hypothesis•Performance on complex tasks can be decomposed into primitive sets of mental operations of the sort revealed in simple laboratory experiments.

•This assumption underlies much of the work on latency in basic neuroimaging analyses.

•Nonetheless there has been very mixed evidence on part-to-whole training.

•This is generally thought to reflect that fact that one cannot use “pure insertion” because the one task will change when you insert another.

•Most sophisticated analyses of reaction time and imaging rely on some sort of “additive factors” where you keep all the processes but just make them more or less difficult.

•Still within the context of a cognitive model one ought to be able to decide when pure insertion will work and when it will not.

Pure Insertion in Space Fortress•Insert Mines or Fortress into game and check brain pattern

•Naïve Insertion: Full Game = Fortress+Mines

•However, this ignores the fact that we are really getting a double count of navigation and visual processing on the right hand side.

-Mines +Mines-Fortress Orbit Mines+Fortress Fortress Both

•Sophisticated Insertion: Orbit+Both=Mines+Fortress

•However, this ignores that there are points in time when one cannot do both fortress & mines in the both condition.

•One needs a model that will perform all of the conditions and look at its predictions for these conditions.

Game structure

• Played 16 games per session – Four of each condition

• 10 sessions 2/day• Sessions 2 and 10 with fMRI– Analyzed the 20s “Maybe mine” segments

Minute 1 3 IFF letters Condition Game Starts Maybe Mine

Minute 2 Maybe Mine Maybe Mine Maybe Mine

Minute 3 Maybe Mine Maybe Mine Maybe Mine

Minute 4 Maybe Mine Game Ends Feedback Fixation

Fixation

Changes from “regular” SF

• Created the separate conditions• Use a keyboard instead of joystick• Fixed timing for mine onsets• No need to choose between points and shots• Fortress vulnerable while mine on screen

0

1000

2000

3000

4000

5000

6000

7000

1 2 3 4 5 6 7 8 9 10Sessions

Poin

ts

Orbit Fortress Mines Both Mines+Fortress Orbit+Both

Points and Point Additivity

•The major point growth is in the fortress conditions•There is a necessary sub-additivity in points

Modeling Plan• Model the well practiced performance

• Consider learning later• Use participants fortress condition for sessions 7-9 as

the basis for the model’s navigating• Run model through all conditions• Compare it to session 9 behavioral data and session

10 fMRI data

Flight data

Keypresses per game

Keypress duration percentages

Scores

• Mine + Fortress = 7373.425 > Orbit + Full = 6959.95

Model total scores

Changes to ACT-R

• Special device for interfacing with the game• Using the Temporal module• Modified – Motor module– Visual module– Imaginal module

Motor module changes• Set motor feature prep time to 0ms– Suggested by Wayne based on Kieras’s 2009 ICCM paper

• Added mostly independent hands– Two execution paths

• Added individual finger queries?finger-check>

left-index busy

• Added new motor actions– Hold-key– Release-key– Release-all-keys– Delayed-punch

Delayed-punch• Holds a finger down for a prespecified time

+manual> isa delayed-punch hand left-or-right finger finger-name {delay time}

• Substitute for a strong visual-motor interface• For the model (mostly) only a few times used– Orbiting 90ms, 125ms, 160ms– Firing 70ms, 90ms, 125ms– IFF 125ms, 160ms– Mine aiming special

Visual features

• Standard text items – foes, conditions, bonus symbols, vlnr count, iff letter

• “Featureless” indicators – no information other than presence or absence – fortress, fortress hit, fortress explosion, ship

explosion, mine, mine explosion, trial over• Ship– Contains most of the critical information

Ship chunkMY-SHIP ISA SHIP SCREEN-POS, VALUE, STATUS, COLOR, HEIGHT, WIDTH, X, Y

ORIENTATION DIST VEL ANGLE VDIR HEX-HIT

MINE-DIST SHOOT-AT-MINE HIT-FORTRESS-FIRST

New visual request

• Attend-and-track– Combines the separate move-attention and start-

tracking requests into one action

New visual buffer

• Visual-search– A search buffer – Makes visicon chunks available for searching

in production matching • Visual-location information only

– Similar to the old LHS !find-location!• Less flexible than a visual-location request

– No :attended, :nearest, or current

• Two primary reasons for adding it– Buffer stuffing not available due to tracking– Allows for modeling control over “stuffing”

• Utility can decide which feature is more important when co-occurrences

(p start-playing =goal> isa goal state start-playing =visual-search> isa sf-visual-location kind ship ?visual> state free ==> +visual> isa attend-and-track screen-pos =visual-search …)

Using the visual-search buffer(p detect-bonus =goal> isa goal check-bonus nil =visual-search> isa sf-visual-location kind new-bonus =imaginal> isa game-state ==> =imaginal> +goal> check-bonus check)

(p detect-hit =visual-search> isa sf-visual-location kind fortress-hit ?imaginal> state free

=imaginal> isa game-state shot-count =count < shot-count 10 counted nil ==> !bind! =new-count (1+ =count) +imaginal> shot-count =new-count counted t)

(p see-mine-onset =goal> isa goal state play =visual-search> isa visual-location kind mine =visual> isa ship

?imaginal> state free … ==> +imaginal-action> isa generic-action action predict-mine-shooting …)

Special Imaginal request• Prediction for shooting the mine

– 250ms, slightly more than a standard imaginal request– Uses existing imaginal-action buffer +imaginal-action> isa generic-action action predict-mine-shooting

• Determines through simulation how to best shoot at the mine given current ship details

MINE-PREDICT ISA MINE-HEADING ACTION TURN FINGER RING DELAY 0.099 DIST 260.8793 HIT-DIST 180.8793

Mine aiming productions(p adjust-mine-heading-with-fortress =goal> isa goal cond2 "+fort“

=imaginal> isa mine-heading action turn finger =finger delay =delay dist =dist

=visual> isa ship <= mine-dist =dist =visual-search> isa visual-location kind mine… ==> +manual> isa delayed-punch hand left finger =finger delay =delay …)

(p reassess-mine =goal> isa goal =visual-search> isa visual-location kind mine

=imaginal> isa mine-heading action drift hit-dist =delay =visual> isa ship < mine-dist =delay … ==> +imaginal-action> Isa generic-action action predict-mine-shooting …)

Model overview

• 77 productions covering eight basic tasks– Encoding pre-trial info and rehearsal– Attend ship then initial thrust and turn to start orbiting– Orbiting the center hex– Shoot at fortress and count hits– Encode and respond to bonuses– Identify mine, aim, and shoot at it– Correct orbiting problems (hit hex or too far out) – Detect ship destruction or trial over

Orbiting productions

• Model tries to stay aimed at the center and orbit clockwise about 95 pixels out

• Broken into groups by ship speed< 1.0 tries to speed up1.0 – 1.7 maintains the normal orbiting > 1.7 tries to slow down

Normal speed (p right-norm-short =goal> isa goal state play dont-turn nil =visual> isa ship <= vel 1.7 >= vel 1.0 >= angle 5 <= angle 10 > vdir 92 ?manual> preparation free ?finger-check> left-index free ==> +manual> isa delayed-punch hand left finger index delay fast)

(p right-norm-default =goal> isa goal state play dont-turn nil =visual> isa ship <= vel 1.7 >= vel 1.0 >= angle 5 <= angle 20 > vdir 88 ?manual> preparation free ?finger-check> left-index free ==> +manual> isa delayed-punch hand left finger index)

(p right-norm-long =goal> isa goal state play dont-turn nil =visual> isa ship <= vel 1.7 >= vel 1.0 >= angle 15 <= angle 30 ?manual> preparation free ?finger-check> left-index free ==> +manual> isa delayed-punch hand left finger index delay slow)

(p thrust-norm-default =goal> isa goal state play =visual> isa ship <= vel 1.7 >= vel 1.0 > dist 93 >= vdir 92 <= angle 2 >= angle -4 ?manual> preparation free ?finger-check> left-middle free left-index free left-ring free ==> +manual> isa delayed-punch hand left finger middle)

(p left-norm-default =goal> isa goal state play dont-turn nil =visual> isa ship <= vel 1.7 >= vel 1.0 < angle -14 ?manual> preparation free ?finger-check> left-ring free left-index free left-middle free ==> +manual> isa delayed-punch hand left finger ring)

Consider the BOLD response• Use the BOLD prediction tools built into ACT-R to

determine activity in the modules• Few changes needed– Made all goal and imaginal modifications in the model

+’s so they’re counted– Added new buffers to track left and right hand actions

separately• Only tracks the execution

– Modified visual tracking so that it is only periodically busy instead of constantly• One attention shift per 2 seconds

-0.20%

0.00%

0.20%

0.40%

0.60%

0.80%

1.00%. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Left Motor Right Motor ACC Parietal Prefrontal Caudate Fusiform

% C

hang

e fr

om B

asel

ine

Orbit Mines Fortress Both

AnteriorPrefrontal

LIPFC

Fusiform

PosteriorParietal

ManualVocal

Auditory

Model’s BOLD motor mismatch

Curious…

Future work

• Address the issues with the model’s BOLD results in the motor system

• Model the performance on the early days and the learning that takes place over days