Problem Solving Protocols

A Tool to Visualize Verbal Protocols A Tool to Visualize Verbal Protocols to Enhance Strategic and to Enhance Strategic and

Metacognitive Abilities in Basic Metacognitive Abilities in Basic Programming Programming

Dr. Carlos Argelio Arévalo MercadoDr. Carlos Argelio Arévalo Mercado

M. en C. Lizeth Itziguery Solano RomoM. en C. Lizeth Itziguery Solano Romo

TAMK Opening IGUAL Project meeting. TAMK Opening IGUAL Project meeting. February 14 – 18, 2011February 14 – 18, 2011

UAA IGUAL Proposed Project StaffUAA IGUAL Proposed Project Staff

Academic Staff Dr. Carlos Argelio Arévalo Mercado (Local Coordinator) M.C. Lizeth Itziguery Solano Romo (Student coordinator and research

support) Dra. Lizbeth Muñoz Andrade (Experiment design and execution) Dra. Laura A. Garza González (Financial support) M.C. Jorge E. Macías Luévano (Infrastructure support)

Student Staff Francisco Javier García Pilar (programmer) Bertha Alicia Cobos (protocol recording) Student 3 (programmer) Student 4 (protocol recording)

2

Aguascalientes.Aguascalientes.

3

Population 1.065.416

Autonomous University of AguascalientesAutonomous University of Aguascalientes

History and Background Different names:

1887 Agriculture School 1871 to 1875 Scientific and Literature Institute 1985 State Science Institute 1906 State High School Before 1973 Autonomous Science and Technology Institute

Since June 19, 1973 Autonomous University of Aguascalientes 36 years since foundation 9 University presidents 54 Under graduate programs, Graduate programs 38, 7 technical

programs, 2 high school programs

4

University PopulationUniversity Population

Professors: 1,985 Students: 14, 310

5

Section Total

Graduate 235

Under graduate 12,456

High School 1,619

Internal UAA Programming StatisticsInternal UAA Programming Statistics

6

Cognitive aspects of programming (Literature Review)Cognitive aspects of programming (Literature Review)

7

Cognitive Aspect Description References Implications

Mental Models

A mental model is an internal representation of a complex system or task, whose construction enables the learner to understand and predict that system's behavior.In the context of programming, this aspect is sometimes called "The Notional Machine"

(George 2000); (Bayman 1983); Johnson-Laird (1983); Winslow (1996); Du Boulay (1989); . (Fixx, Wiedenbeck, y Scholtz 1993; Hegarty 1993; Ma et al. 2007; Ramalingam 2004)

Valid mental models have to be taught explicitly.

Problem occurs when novice creates an invalid mental model.

Schemas and Strategies

A strategy (or plan, or cliché) is a predefined solution within the programmers knowledge structure, used when a recurring problem pattern is recognized.Writing a program is not a straightforward process. It is incremental and hypothesis driven.A key part of a program (called focal point,) is used by an expert, and the solution is built around that point.

(Fixx, Wiedenbeck, y Scholtz 1993); Brooks (1990); Rist (1995, 514); Soloway y Ehrlich (1984); Davies (1993)

Rist…

Strategic knowledge of program construction is hard to teach, and not sufficiently emphasized in traditional courses.Not enough instruction is given to novices on how to "put the pieces together“.Text books often only show the final state of a problem/program, but not the process of construction.

Metacognition

Metacognition is defined as "awareness about how a person learns or solves a problem". Flavell (1979)Through Metacognition, an individual can define the nature of a problem, select a useful mental representation, select an effective strategy, monitor the progress of problem resolution and adjust strategies during the task.

Flavell (1979); Gourgey (1998); Williams y Upchurch (2001) ; Roth (2004),

Metacognitive aspects of program design are included within strategic knowledge.

These Metacognitive knowledge must also be taught explicitly

Instructional aspects related to programmingInstructional aspects related to programming

8

Instructional Aspect

Description References Implications

Learning Style According to a previously selected model, an individual can have different preferences or characteristics that facilitate his/her learning process

Kolb (1984)Felder (1996)(Briggs y Myers, 1995)

There are no statistically significant evidence about some particular learning style being a predictor of success in the first programming course.

Previous Experience

A previous programming experience can occur in the context of high school education

(Fauxx 2006); (Hagan y Markham 2000); (Holden y Weeden 2005).

There is a positive effect on performance if the student has had a previous programming experience.

This positive effect tends to disappear following undergraduate semesters.

Self-Efficacy How well one can execute courses of actionrequired to deal with prospective situations

(Bandura 1982); (Brosnan 1998); (Byrne 2001); (Ramalingam 2004); (Heggestad 2005)

It has been identified as a positive influence on first programming course performance.

Some have stated that self-efficacy its just a residual variable of previous course performances.

Math ProficiencyThe abstract and logical reasoning ability required to solve mathematical problems, is similar to the ability required to program computers

(Byrne 2001); (Hu 2006); (White 2003); (Jenkins 2002)

Has a strong correlation with positive performance.

But ethical and practical curricular issues arise.

9

Components of an information system design theory Components of an information system design theory (ISDT) (Adapted from Walls, et. al, 1992)(ISDT) (Adapted from Walls, et. al, 1992)

1. Meta requirements Describes the class of goals to wich the theory applies2. Meta-design Describes a class of artifacts hypothesized to meet the meta-requirements3. Kernel theories Theories from natural or social sciences governing design requirements4. Testable design products hypothesis Used to test whether the meta-design satisfies the meta-requirements

1. Design method A description of procedure(s) for artifact construction.2. Kernel theories Theories from natural or social sciences goverment design process itself.

3. Testable design process hypotheses

Design Product

Used to verify wether the design method results in an artifact wich is consistent with the meta-design

Design Process

Components of an information system design theory (ISDT)

Doctoral Research General ObjectiveDoctoral Research General Objective

Apply the ISDT framework to produce a design theory that outlines a kind of software artifact whose properties include: visualizations of valid mental programming models, strategic problem solving knowledge and Metacognitive reasoning of programming experts on the act of writing programs, to foster meaningful learning and better knowledge transfer to novice programming students.

10

Specific ObjectivesSpecific Objectives

Propose an Information Systems Design Theory for a kind of software artifacts that promotes the transfer of valid programming mental models, problem solving strategies and Metacognitive abilities to novice students.

Design and build de artifact.

Empirically test the effectiveness of the artifact.

11

ISDT Design Product Kernel TheoriesISDT Design Product Kernel Theories

12

Kernel Theories

Description Evidence / Principles References

Dual Coding Theory

When information is registered both verbally and visually, it's said that be "Dually Coded". Information is better retrieved if it is dually stored, because if one part is lost (visual or verbal) the other is available and can be retrieved.

Empirical data suggests that students using study materials designed to promote dual coding, seem to have better performance

(Kuo y Hooper 2004; Meyer y Sims 1994; Sadoski y Paivio 2004)

Verbal Protocols

Verbal protocols where developed to study the short termmemory process of people (to what things the pay attention and in what order) in the process of solving a problem.

They document the mental behavior of a person, by asking him or her to "Think aloud" while doing a certain task

Verbal data are a valid source of comprehension and study of human cognitive processes.A phrase is the representation of an idea and can be used to identify different kinds of information y different types of cognitive processes.Used in the context of basic programming to analyze common solution patterns.Also used in usability studies.

Ericsson y Simon (1980), Nielsen, Clemmensen, y Carsten (2002), (Détienne 1995; Burkhardt, Détienne, y Wiedenbeck 1997; Rist 1996; Rist 1989);

Worked examples and

completion problem

effect

Worked examples can facilitate the construction of schemas and strategy transfer more efficiently than solving the problems themselves. This prediction is explainedin terms of "cognitive load reduction".It has the limitation that worked examples do not force low motivated students to study them. Worked examples can be complemented with the "completion problem effect" pedagogical strategy.

Empirical studies have demonstrated the effectiveness of worked examples in fieldssuch as algebra, and heuristic knowledge domains.Important: they should be used only during the first stages of learning, to avoidthe so called "expertise reversal effect" and they should not have to be used excessively to avoid the transfer of too much stereotyped solution patterns

(J. Sweller, van Merrienboer, y Paas 1998; John Sweller 1994); (Gerjets, Scheiter, y Catrambone 2004; Renkl, Hilbert, y Schworm 2009; Renkl, Atkinson, y Große 2004; M. Ward y J. Sweller 1990; J. Sweller y G. Cooper 1985)

Verbal Protocols editing and recording processVerbal Protocols editing and recording process

13

Select and Categorize a problem

(instructor)

Record the problem solving protocol for the

selected problem(Programmer &

Interviewer)

Transcribe and Edit the protocol in the systemAs a Worked Example

(Instructor)

Add Visual Aids(Animated Code)

(Instructor)

Add additional notes to the protocol(i.e Focal Point, general strategy)

(Instructor)

Visualize and Study theProblem Solving Protocol

(Student)

Development of the ArtifactDevelopment of the Artifact

14

Evolution of the user interfaceEvolution of the user interface

15

FlashFlash

AccessAccess

Web (PHP)Web (PHP)

User interface for verbal protocol visualization User interface for verbal protocol visualization

16

Protocol Verbalization

Dual Coding

Programming Problem

Instructor aditional Notes

Protocol Navigation

Modified user interfaceModified user interface

17

Problem search by keywords

The interface was subject to several tests to refine its usability. For example a feature was added to allow a student to “jump” directly to a specific step of the protocol

18

Instructor annotations(i.e. focal point)

Animated code

Go to a specific protocol step

Navegación

Verbalizations

Modified user interfaceModified user interface

Measurement instrumentMeasurement instrument

19

To measure the tool’s capability to transfer strategic knowledge to novice undergraduate programmers, we designed a standard test, consisting of three basic programming problems, to assess the student’s skills

Cuasi-experimental conditionsCuasi-experimental conditions

To test the effectiveness of the tool, a semi-experimental setting was designed, using two groups of programming students: one from 2nd semester Computer Science students, and another from 2nd semester Electronic Engineering students, both

from Autonomous University of Aguascalientes (UAA), México. Selection of participants was not random. Complete groups were invited,

given that the experiment was conducted during a period of normal classes. The Electronic Engineering group (n=20) was selected as the control group

(TRAD). The computer science group (n=18) was to serve as the experimental group (EXP). 55% of the (TRAD) group had previous programming experience from high school, while 41% of the (EXP) group had previous programming experience.

Two days before the application of the test, four exercises were given to both groups for them to study and practice. This exercises were the same ones loaded in the Protocol Visualizer Tool.

20


Controlled conditions for both groups were: Explanatory lecture sessions had a one hour duration for both groups. The given time to answer the instrument was limited to an hour for both

groups. A “motivation” factor was introduced for both groups in form of “extra

points”. Characteristics of the lab computers used from both groups were the

same Previous programming experience was similar in both groups (55% for

TRAD and 41% for EXP). At the time of the experiment, the instructional content given (during

normal classes) to both groups was the same. Both groups where studying basic data structures in C Language.

21


Uncontrolled conditions were: Teachers from both groups were different, but they came from the same

programming academy, taught the same content, and had the same experience in teaching.

It was not possible to record the individual answering time of the test participants.

Selection of participants was not random. In both cases, complete groups where invited to participate in the study.

22

ResultsResults

23

Statistical dataStatistical data

24

Correlation AnalysisCorrelation Analysis

25

Limitations of the studyLimitations of the study

Given the uncontrolled conditions reported, the positive results obtained by the EXP group using the Protocol Visualizer Tool, cannot be generalized to be valid for all the population of first year undergraduate programming students.

An uncontrolled variable was teaching style: that is, even though both teachers were part of the same academy and had similar background and experience, the two groups having different teachers could have had an unmeasured effect on the final results.

Also, duration of the actual study was limited to three days, given the limited availability of the students who voluntarily participated.

26

Current state: Interface RedesignCurrent state: Interface Redesign

27


28


29


30

Current stateCurrent state

Protocols being recorded by social service students

Problem selection Quality control Editing Transcription / System mounting

31

Future activitiesFuture activities

Future studies need to be longitudinal in nature, so that the effect of a longer exposure of the students to the tool can be measured.

A randomly selected sample of participants is desirable, but this kind of scenario is not always possible (or practical) given the nature of every day lectures.

It is also possible to load the tool with protocols that involve problems related to other programming languages or paradigms such as Java, C#.

Also, the graphical user interface can still be improved through usability test, in order to use with other Web-enabled platforms (such as mobile devices).

Extend the functionality of the tool, by adding pedagogical features such as completion problems, in selected segments of the protocols.

Combine the protocol visualizer with other learning technologies were possible.

32

Documents

Problem Solving Protocols