An Experiment in Students’ Acquisition of Problem Solving Skill from Goal-Oriented Instructions

AN EXPERIMENT IN STUDENTS’ ACQUISITION OF PROBLEM SOLVING

SKILL FROM GOAL-ORIENTED

INSTRUCTIONSMatej Guid1, Jana Krivec2, Ivan Bratko1

Artificial Intelligence LaboratoryFaculty of Computer and Information Science, University of Ljubljana, Slovenia

and

Department of Intelligent SystemsJožef Stefan Institute, Ljubljana, Slovenia

COGNITIVE 2012

OUTLINE

Part IChunks in declarative knowledge

vs Chunks in procedural knowledge

Part IIConceptualization of domain knowledge:how the goal-oriented instructions were obtained

Part IIIThe experiment in students’ acquisition of problem solving skill from goal-oriented Instructions

CHUNKS IN DECLARATIVE KNOWLEDGE

CHUNKS IN DECLARATIVE KNOWLEDGE• completed units of logically related information clusters• facilitate retrieval and use• better utilization of limited working memory capacity

See details in Gobet F. et al. Chunking mechanisms in human learning . Trends in Cognitive Sciences, 2001.

real-gameposition

randomlyshuffled pieces

reconstructing chess positionsGobet et al. (2001)

pioneer works of De Groot, Simon, Chase, Gobet etc. used chess as a domain of choice...

CHUNKS IN DECLARATIVE KNOWLEDGE

CHUNKS IN DECLARATIVE KNOWLEDGE• completed units of logically related information clusters• facilitate retrieval and use• better utilization of limited working memory capacity

COMMONLY ACCEPTED

pioneer works of De Groot, Simon, Chase, Gobet etc. used chess as a domain of choice...

See details in Gobet F. et al. Chunking mechanisms in human learning . Trends in Cognitive Sciences, 2001.

LETTERS OR WORDS?

chunks in declarative knowledge

egdelwonk evitaralced ni sknuhc

familiar placements of groups of pieces

CHREST (Chunk Hierarchy and Retrieval STructures)

&domain of chess

WHAT ABOUT CHUNKS IN PROCEDURAL KNOWLEDGE?

CHUNKS IN PROCEDURAL KNOWLEDGEWe attempt to:• show the existence of chunks in procedural knowledge, • define chunks in procedural knowledge – “procedural chunks”• see how they are incorporated in ones memory.

X LITTLE KNOWN

People learn procedural knowledge by (sub-consciously)constructing meaningful units of procedural knowledge.

Krivec J., Guid M. & Bratko I. Identification and Characteristic Descriptions of Procedural Chunks. COGNITIVE 2009.

Our previous study: reconstructing chess games Krivec, Guid & Bratko (2001)

Current work: chess beginners were playing chess games against a computer

A PROCEDURAL CHUNK IN CHESS• a sequence of chess moves that all together belong to a chess concept• memorized by a player as a whole.

HOW COULD A WORLD CHAMPION BLUNDER A MATE IN ONE?

Chessbase.com: How could Kramnik overlook the mate? http://www.chessbase.com/newsdetail.asp?newsid=3512

Kramnik in the press conference: "It was actually not only about the last move. I was calculating this line very long in advance, and then recalculating. ........ I was feeling well, I was playing well, I think I was pretty much better. I calculated the line many, many times, rechecking myself... It's just very strange, I cannot explain it."

Deep Fritz – KramnikMan vs Machine, Bonn 27.11.2006

A rare pattern with white knight on f8 and black king on h8?

position after 34... Qa7-e3, allowing 35.Qe4-h7#

If the last two moves were 1.Ng6-f8++ Kh7-h8,wouldn’t it be “impossible” not to see 2.Qe4-h7#?

THE MECHANISM OF “EINSTELLUNG” EFFECT

Bilalić B. et al. The Mechanism of Einstellung (Set) Effect. Current Directions of Psychological Science, 2010.

EXAMPLE: THE EINSTELLUNG EFFECT IN CHESS EXPERTS Bilalić, McLoud, Gobet (2010)

expert chess players were asked to find the shortest solution:

1st solution: smothered mate (5 moves)2nd solution: less familiar (3 moves)

smothered mate (before: 1st solution)not possible anymore!

PROCEDURAL CHUNKS IN CHESS

A “simple” solution...


1. Qe6+ Kh8 2. Nf7+ Kg8 3. Nh6++ Kh8 4. Qg8+ Rxg8 5. Nf7#




















… and a “difficult” solution


1. Qe6+ Kh8 2. Qh6


1. Qe6+ Kh8 2. Qh6


1. Qe6+ Kh8 2. Qh6


1. Qe6+ Kh8 2. Qh6


What makes the difference in cognitive difficulty of finding the two solutions?

“easy” “difficult”

CONCEPTUALIZATION OF PROCEDURAL KNOWLEDGE

ORIGINAL THEORY PROBLEM SOLUTION.......................................................................

axiomslawsformulasrules of the game…

path: requires excessive computation, difficult to memorize

CONCEPTUALIZEDDOMAIN THEORY

DECLARATIVE KNOWLEDGE PROCEDURAL KNOWLEDGE

WHAT? HOW?basic domain knowledge goal-oriented rules

Možina M. et al. Goal-oriented conceptualization of procedural knowledge. Conference on Intelligent Tutoring Systems, 2012.







WHAT? HOW?basic domain knowledge goal-oriented rules







basic rules of chesspiece movementsthe 50-move rule…the “right” corner conceptbasic strategy…

procedures: IF-THEN rulessimple and compact ruleseasy to memorize…intuitive knowledgeintermediate goals…



PROBLEM STATE SPACE

::

::

::

. . .

. . .

start node

goal nodes

(too) longsolution path

. . .

LEARNING INTERMEDIATE GOALS

::

::

::

. . .

. . .

goal nodes ofintermediate goals

start nodes ofintermediate goals

. . .

CONCEPTUALIZATION OF DOMAIN KNOWLEDGE: CHESS ENDGAME

goal-oriented instructions

example games with goal-oriented instructions

KBNK – the most difficult of elementary chess endgames:several recorded cases when even grandmasters failed to win

the result of conceptualization: Hierarchy of (only) 11 GOALShttp://www.ailab.si/matej/KBNK/

TEACHING MATERIALS (1): TEXTBOOK INSTRUCTIONS

TEACHING MATERIALS (2): EXAMPLE GAMES WITH GOALS

A GRANDMASTER FAILED TO WIN ...

A grandmaster of chess failed to win the following endgame…


GM Kempinski (white) – GM Epishin (black), Bundesliga 2001

“wrong corner”

“right corner”


















































… but why our students didn’t?

INTERMEDIATE GOAL: BUILD A BARRIER








Black established a barrier that prevents the white king from reaching the wrong corner.

THE GOALS OF THE EXPERIMENT

We were interested in two questions:

1. How useful are the goal-oriented instructions to the student as a help towards mastering the play in this domain?

2. What was the form of the student’s “internal representation” of the acquired knowledge?

Our hypothesis: • a goal-based representation • similar goal structure as in the instructions.

EXPERIMENTAL SETUP

PARTICIPANTS• three chess beginners of different strengths

TEACHING MATERIALS• goal-oriented textbook instructions• example games with instructions

TIME LIMIT• 10 minutes per game

START POSITIONS• mate-in-30 moves or more assuming optimal play

ENVIRONMENT• Fritz 13 chess software by Chessbase• by using chess tablebases computer defended optimally• moves and times spent for each move were recorded automatically

OTHER CONSIDERATIONS• the experiment was divided into three phases

Student 1: Class B player (best) Student 2: Class C playerStudent 3: Class D player (worst)

THE EXPERIMENT

Phase II: Examination of teaching materialsparticipants were given access to teaching materials (no more than 30 minutes)

Phase III: Playouts against optimally defending computerafter each game (but not during the games!) the materials were accessible to participants

The students learned the skill operationally in up to an hour’s time of studyingthe instructions and testing their skill in actual problem solving (playing the endgame).

Phase I: Three trial KBNK gamesparticipants were unable to deliver checkmate

DTMdev = DTMplayed - DTMoptimal

IDENTIFICATION OF PROCEDURAL CHUNKS: METHOD

• Chase and Simon (1973): longer time interval during the reconstruction of a meaningful unit of material reveals the recall of a new structure/chunk from the long-term memory.

• Bratko, Tancig & Tancig (1984): “collective reconstruction”

1. For all moves in each game:calculate an average value and SD of normalized time medians (use z values)

2. Consider all the moves that exceeded the boundary of the average value plus one SD as a “long time interval” (and as such candidates for the beginning of a new procedural chunk)

3. If such a candidate appeared in the majority of the games, consider it as the beginning of a procedural chunk.

BACKGROUND IDEAS

OUR APPROACH

IDENTIFICATION OF PROCEDURAL CHUNKS: RESULTS

Automatically detected procedural chunks in the students’ games corresponded almost perfectly to the goal-oriented rules in the textbook instructions.

Detected chunks in the final phase of the endgame

(see paper for detailed descriptions)

Phase III: delivering checkmate, once a “barrier” is set up

ACQUIRING PROCEDURAL CHUNKS DURING LEARNING

the number of different chunks appearing in consecutive games played by a student

CONCLUSIONS & FUTURE WORK

1. How useful are the goal-oriented instructions to the student as a help towards mastering the play in this domain?

2. What was the form of the student’s “internal representation” of the acquired knowledge?

Automatically detected procedural chunks in the students’ games corresponded almost perfectly to the goal-oriented rules in the textbook instructions. We also measured the dynamics of acquiring these chunks during the learning time.

The students learned the skill operationally in up to an hour’s time of studying the instructions and testing their skill in actual problem solving (playing the endgame).

We intend to strengthen these experimental results by scaling up the experiments in terms of the number of subjects, and by extending the experiments to other domains (other chess tasks and domains other than chess).

? http://www.ailab.si/matej/

dr. Matej Guid. Artificial Intelligence Laboratory,Faculty of Computer and Information Science, University of Ljubljana. Web page: http://www.ailab.si/matej

Documents

An Experiment in Students’ Acquisition of Problem Solving Skill from Goal-Oriented Instructions