Human and machine diagnosis of scientific problem-solving abilities

JOURNAL OF RESEARCH IN SCIENCE TEACHING VOL. 23, NO. 4, PP. 26S275 (1986)

HUMAN AND MACHINE DIAGNOSIS OF SCIENTIFIC PROBLEM- SOLVING ABILITIES

RON GOOD

Department of Science Education, Florida State University, Tallahassee, Florida 32306

ROBERT KROMHOUT

Department of Physics, Florida State University, Tallahassee, Florida 32306

WYLLIS BANDLER

Department of Computer Science, Florida State University, Tallahassee, Florida 32306

Abstract

Diagnosis of the problem-solving state of a novice student in science, by an accomplished teacher, is studied in order to build a computer system that will simulate the process. Although such “expert” systems have been successfully developed in medicine (MYCIN, INTERNIST/ CADUCEUS), very little has been accomplished in science education, even though there is a reasonably close parallel between expert medical diagnosis of patients with physiological problems and expert instructional diagnosis of students with learning problems. The system described in this paper, DIPS: Diagnosis for Instruction in Problem Solving, involves a new line of research for science educators interested in interdisciplinary efforts and ways in which computer technology might be used to better understand how to improve science learning. The basic architecture of the DIPS system is outlined and explained in terms of instruction and research implications, and the role of such “intelligent” computer systems in science education of the future is considered.

Preface

Science education research is primarily concerned with how to best help students learn science. A perusal of back issues of JRST supports this statement and helps one to gain a sense of history of science education research. The mostly psychometric research of the 1970s continues into the 1980s, but with an indication that different, perhaps more “basic, ” research techniques are gaining in acceptance among research- minded science educators. Naturalistic or ethnographic methods, while not as yet

0 1986 by the National Association for Research in Science Teaching Published by John Wiley & Sons, Inc. CCC 0022-4308/86/040263-13$04.00

264 GOOD, KROMHOUT, AND BANDLER

widely used in published reports, seem to have been accepted by most NARST members as capable of serving a useful purpose.

This paper represent a form of research that utilizes the expertise of a number of different specialists, including science educators, computer scientists, and others who fall, in part at least, under the general category of cognitive scientist. How the role of the science education researcher evolves in this group-oriented type of research will depend on many factors, including the interest and acceptance by NARST-type researchers. That machine-mediated learning will become an increasingly important part of science education is not the real issue; the important question to ask is, How will NARST-type science educators influence the process?

Introduction

The main purpose of this position/research paper is to describe a line of research that could become increasingly important to science education researchers. Tradition- ally, science education research has been done by persons within the somewhat vague boundaries of our discipline. This paper describes a line of research that suggests a new role for some science education rersearchers, a role that requires a cooperative, interdisciplinary approach. Development of the research team is described and the project itself is set forth in some detail, with implications for using the product of the first phase of the research to conduct expanded science education research. One of the promising features of this line of research is the establishment of a large database on methods of scientific problem solving that will allow for more consistency in data handling and interpretation of in-depth interviews with subjects. The procedures students use to solve science problems and their progress in this area can be tracked with resulting implications for the design of future curricula and instruction.

Problem-Solving Research

Human diagnosis of scientific problem-solving abilities can be traced back for many years, but for our purposes in this paper we will use 1972 as the beginning of this type of work. In that year Newell and Simon’s, Human Probtem Solving appeared and served as a guide and stimulus to the more recent work designed to learn how novices and experts proceed in attempting to solve well-defined problems in science, particularly in physics. Newell and Simon studied the procedures used by adults to search for solutions to problems with low semantic requirements, such as cryptarith- metic and Tower of Hanoi puzzles. Their research method required the subject to think aloud while trying to search for a solution to the problem, thus providing a trace of the solution path that could later be analyzed in detail. Their study of the procedures used by people to solve problems with high structure and low semantic content was soon followed by similar efforts using content in mathematics and science that required considerably more knowledge related to the problem. An account of this type of research, in solving mechanics problems in physics, that received widespread attention in the scientific community because of its publication in Science, was reported by Larkin et al. (1980). Other reports of studies using think-aloud techniques and clinical interviews for related physics content include: Simon & Simon (1978), Champagne, Hopfer, and Anderson (1980), McCloskey (1980), Trowbridge and McDermott (1980, 1981), Chi, Fletovich and Glaser (1981), Larkin (1981), Lugar (1981), Clemont (1982),

SCIENTIFIC PROBLEM-SOLVING ABILITIES 265

diSessa (1982), Reif and Heller (1982), White (1983), Aguirre and Erickson (1984), Anzai and Yokoyama (1984). Other closely-related studies of scientific problem solving, but not restricted to physics, include: Gagne (1966), Greeno (1976), Preece (1976), Clement (1979), Gunstone (1981), Stewart (1982), Smith and Good (1984). A recent paper by McDermott (1984) summarizes much of the problem-solving research in mechanics, the area of physics that receives the greatest attention in high school and introductory college courses.

Most of this research is based on an information-processing model of cognition. An article by Case in the 1980 AETS Yearbook, a paper by Stewart and Atkins (1982), and a symposium organized by Good et al. at the 1983 NARST meeting have helped to focus the attention of science education researchers on this relatively recent model of cognition.

Expert Systems Research

Related to the expert-novice research on scientific problem solving is the rapidly growing field of expert computer systems research. Knowledge generated within the field of artificial intelligence (AI) is combined with procedural knowledge elucidation techniques made well known in Human Problem Solving by Newel1 and Simon (1972) and recently refined in a book by Ericsson and Simon (1984). Development of early expert computer systems concentrated mainly on medical diagnosis, although other systems are well known within A1 circles, such as DENDRAL, a system that analyzes mass spectrographic and other chemistry experiment data (see Buchanan & Feigen- baum, 1978); PROSPECTOR, a system that determines the most plausible mineral deposit locations (see Duda, Gaschnig, & Hart, 1979); HEARSAY-11, a speech understanding program (see Erman, et al., 1980); and EXPERT, a system for developing consultation models (see Weiss & Kulikowski, 1979).

The fundamental idea of expert systems is that of manipulating large amounts of knowledge gained from experts, say physicians, into a form that can be used to solve problems. In medical expert systems such as MYCIN (Shortliffe, 1976) and IN- TERNIST (now called CADUCEUS, Pople, 1977) the expert physician is carefully studied during diagnosis and is later questioned to reveal how and why things are done. A computer system is developed with the intent of having it simulate the expert, thus the phrase, “expert system.” CADUCEUS, for example, displays expert performance in about 85% of internal medicine, which means that it has a very large knowledge base.

Hayes-Roth, Waterman, and Lenat (1983) list 10 categories of expert systems applications: (1) interpreting sensor data, (2) predicting likely outcomes, (3) diagnosing system malfunctions, (4) designing systems, (5) designing actions, (6) monitoring, (7) debugging or prescribing remedies, (8) repairing, (9) instructing (diagnosing, debugging, and repairing learning behavior), and (10) controlling system behaviors (p. 14). Instruction systems include diagnosing, debugging, and repairing student behaviors. In Intelligent Tutoring Systems, Sleeman and Brown (1982) describe important aspects of computer systems that can be used to assist student learning. The success achieved by medical diagnostic expert systems such as MYCIN and CADUCEUS and the beginnings of expert instructional systems described by Sleeman and Brown (1982) suggest a bright future for expert systems in assisting teachers and students.


The assumption that there is a reasonably close parallel between expert medical diagnosis of patients with physiological problems (and detectable symptoms) and expert instructional diagnosis of students with learning problems (and detectable knowledge and reasoning deficiencies) forms an important part of the basis of the project described in this paper. This assumption is supported by Reif (1983) who stated, “Medicine (like education) is an applied field aiming to achieve a transfor- mation process of the form Si-Sf where a system S (the ‘patient’) is transformed from an initial state where S is sick to a final state where S is functioning normally. The theoretical issues to be addressed in medicine are then correspondingly similar”

For a detailed account of expert systems see Building Expert Systems by Hayes- Roth, Waterman, and Lenat (1983). We close this section on expert systems by referring the reader to their Figure 1, the general structure of what Hayes-Roth, Waterman, and Lenat (1983) refer to as an “ideal” expert system (p. 17). This structure can then be compared to the basic structure of the expert system described in this paper-DIPS, Diagnosis for Instruction in Problem Solving.

(P. 3).

DIPS: Diagnosis for Znstruction in Problem Solving

In this section we present a brief history of the research project, identify its main goals, describe progress since early 1984, and discuss future plans.

A Brief History

This information is included mainly to provide a glimpse of the development of a loosely-knit, interdisciplinary group of faculty into an institute established for research purposes. Early in 1982, a group of six faculty members at Florida State University began regular meetings to discuss cognitive science issues. Each faculty member was from a different department: Darryl Bruce, psychology; Robert Gagne, instructional Design; Ron Good (co-organizer), Curriculum and Instruction; Saakko Hintikka, Philosophy; Abe Kandel, Computer Science; Robert Kromhout (co-organizer), Physics. The “Cognitive Science Study Group” met regularly throughout 1982 and by the end of 1983 the number of faculty members had more than doubled and its coordinators, Good and Kromhout, had attended two annual meeetings of the Cognitive Science Society. Interest had now focused on a small number of issues, with scientific problem solving among them. With the addition of Wyllis Bandler to the faculty and his immediate interest in the Study Group, expert systems research became an important focus within the regular meetings. An “interest group” com- posed of Bandler, Good, Kandel, and Kromhout began weekly meetings and developed a proposal to build an expert system that would stimulate the diagnostic abilities of an accomplished human physics teacher. They decided to concentrate on diagnosis rather than remediation since that step seemed to be conspicuously absent in most instructional settings, in spite of its obvious importance.

The Cognitive Science Study Group soon became an Institute for Cognitive Sci- ences and a research proposal was finalized and sent to NSF early in 1984. The smaller interest group continued its regular meetings as did the larger Institute (by early 1984, about 20 members). Good and Kromhout continued as co-directors of the Institute and Bandler assumed a central role in the development of the expert system, DIPS.


These three faculty and a contingent of graduate students, most notably Sue Szabo, worked to define the project and develop a satisfactory model. Although lack of funding kept progress slow, work continued and is described in the following sections.

DIPS Project Goals

The most obvious goal of the project is to develop an expert computer system that closely simulates a human expert in diagnosing the difficulties of a student who is trying to solve certain science problems, in this case mechanics problems in physics. Although accurate diagnosis is obviously an important part of an effective instructional system, it is only during a one-to-one help session that a teacher has a real opportunity to accurately diagnose obstacles to learning. It is just as difficult for a teacher to accurately diagnose learning deficiencies/obstacles during a session with 25-30 students as it would be for a physician to accurately diagnose the causes of illnesses while conducting a class with 25-30 patients. Since the typical classroom teacher does not have the time to do very much individual diagnosing, except for paper-and- pencil exams and occasional after class discussions, an ‘‘intelligent’ ’ computer system could provide important guidance for the teacher and the student. If accurate diagnoses were available, there would be a much better chance of helping students overcome common obstacles such as the pre-Newtonian or Aristotelian conceptions of motion and forces, velocity-acceleration confusion, translating problems from natural lan- guage to mathematical notation, etc.

In addition to goals related to helping students become better problem solvers in physics or other science areas, we see DIPS as a research tool that has vast potential. Record-keeping abilities promise a data base of students’ learning difficulties that could go a long way toward establishing a science of science learning. It would be an extension of the recent use of naturalistic or ethnographic methods in science education, but it would have the important advantage of using a common data base and consistency in data handling.

In summary, DIPS project efforts are designed to answer the following questions:

(1) How does an accomplished science teacher diagnose the problem-solving ability of a

(2) Can this human diagnosis be simulated by a machine? (3) How can computer record-keeping abilities of an expert computer system add to our

student during an individual help session?

database on (and understanding of) scientific problem solving?

Human Diagnosis of Problem-Solving Abilities

How does an accomplished science teacher diagnose the problem-solving abilities of a student? This apparently simple question has led to many hours of discussion and analysis of actual diagnostic sessions by members of the DIPS project staff. Because physics problem solving has been studied more extensively than other types of problem solving, and because of the academic backgrounds and teaching experiences of two DIPS group members, classical mechanics was selected as the content area to study first. Diagnostic sessions with high school and beginning college students (novices) led to agreement about common techniques used by the teacher (diagnoser) and common responses of students. A diagnostic session with a physics student usually begins


with the student’s own perceptions of his difficulties, i.e., a self-diagnosis. Self- diagnosis is seldom accurate, but, properly interpreted, does yield information about the state of the student.

The immediate next step is usually the posing of a problem in the area of the student’s most recent or acute difficulty. Often this problem is chosen from a text, but may be posed by the instructor. The student is encouraged to begin analyzing the problem, but is guided into beginning with very basic principles and the use of dia- grams and explicit modeling with explicit assumptions. Prompts are used sparingly to stimulate a student to verbalize “hangups” or “sticking points.” Nonverbal clues such as facial expressions are sometimes useful in trying to determine the degree of a student’s understanding. Usually before the problem has been completed. auxiliary or related problems are posed and nontraditional conceptual puzzles and questions are used to be sure simple algorithmic solutions are not mistaken for a genuine understanding of underlying concepts and principles. Examples of nontraditional problems in physics, that are used to assess understanding, have been reported by Clement (1982), diSessa (1982), White (1983), McDermott (1984), and others. The problems usually involve qualitative descriptions of simple systems, often accompanied by dia- grams, and the student is asked to predict and explain certain events.

To summarize, the following procedures are used to determine what human diag- nosers do when they attempt to diagnose the problem-solving state of a student:

(1) Discuss with the experts the general diagnostic procedures they use in trying to help

(2) Videotape diagnostic sessions and discuss with the expert what was occurring and

(3) Determine what seems to be the content knowledge and heuristics needed to solve

(4) Analyze the procedures used by experts as they attempt to solve science problems. (5) Analyze the procedures used by novices as they attempt to solve science problems.

students on an individual basis.

why.

specific science problems.

Machine Diagnosis of Problem-Solving Activities

MYCIN, CADUCEUS, and other machines (computer programs) have shown that they can diagnose patients’ symptoms and there is no reason why similar research cannot be done with students’ problem-solving abilities. While much is yet to be learned about human problem solving, a good foothold has been established in terms of useful research techniques and data interpretation. Accurate diagnosis for effective instruction in problem solving is just as important to the educator as accurate diagnosis for medical remediation is important to the physician.

Machine simulation of human diagnosis of a student’s problem-solving state requires a large repertoire of appropriate questions and problems to pose and the knowledge to interpret the student’s responses correctly. Some of the characteristics of the human diagnostic process can be simulated by an expert computer system, some can be substituted for with only slight loss of replication, and some, such as observation of nonverbal clues, can be replaced only by relatively crude techniques such as ques- tioning the student’s self-perceptions. The efficient use of the student’s and the machine’s time strongly suggests that the system pose a problem whose area and difficulty are based on answers to specific self-analysis questions, that the system move fairly


quickly through a sequence of related problems of increasing or declining complexity on the basis of a few carefully chosen questions per problem until conceptual and/or skill difficulty areas are pin-pointed. Only at this point would a more searching diagnostic process begin to verify the area of difficulty, to distinguish global misconceptions from errors of definition or lack of skills, and to check related concepts not tested in the preliminary diagnostic, but known to be often associated with detected conceptual difficulties.

The knowledge base needed for this diagnostic procedure has a relatively accessible component: the problems to be used, the questions posed, and (accessible with somewhat more difficulty) suggestions for successive moves to new problems as suggested by various predicted answers, as well as clues to diagnosis at this preliminary stage. The time required to build up such a base will be large but can be distributed among experienced teachers.

The knowledge base for the more searching (“thought path”) diagnostic procedure must be obtained more subtly. Techniques that have been, and will be, used include:

(1) Taping of actual student-teacher diagnostic sessions with ex post fucfo self-analysis of the teacheddiagnostician’s behavior.

(2 ) Recording of problem solving by experts with running commentary on previously observed student difficulties and perceived misconceptions.

(3) Specific diagnostic procedures suggested by experts reacting to observed student paths in pilot tests of the preliminary diagnostic system.

(4) Specific improvements suggested by teachers during field testing and even later during use of the system, as well as improvements suggested by the data collected in the archival memory.

DIPS Design

By analyzing diagnostic sessions between teachers (experts) and students and through many discussions of the nature of diagnosis for instruction in problem solving, DIPS project staff agreed on the basic design of the system. This design, worked out in final form by S. Szabo with help from M. Dawson, is shown in Figures 1, 2, and 3.

Figure 1 shows the three “zones” of the system. These three zones are: (1) the communicator, (2) the diagnoser, and (3) the archives. Each zone has its typical users; students deal directly with the communicator while both teachers and researchers converse with the archives. The problem area experts (initially, physics teachers) put material into the diagnoser, giving content to the stores of questions and method to the inference mechanism.

The diagnoser (see Fig. 2) will consist of two main subzones: the chain-of-thought analyzer and the cube processor. The former will trace, on selected question sequences, the path of the student’s reasoning, and individuate any departures from “model” paths. It will be the most delicate and subtle portion of the entire system, and will be developed last, relying as it does heavily on correct prior operation of the other subzone, and on feedback information from the archives of the first prototype to be put into the field.

The cube processor is the principal subzone of the diagnoser in the preliminary model, and will remain of great importance in the final version. It consists of a very


I C A W R ZONE a DIAGNOSER ZONE

Fig. 1 . DIPS design showing “zones” of opertions.

extensive and highly organized knowledge base together with its own controller guid- ing the trajectory of the interaction between the user and the base. The base consists of a number of cubes (actually hypercubes), each dealing with a single topic within the chosen domain of expertise, for example. friction between blocks and planes, within the general domain of high school physics. Each block contains many (ideally 625) problems about which there will be appropriate questions, arranged according to their difficulty along four axes: conceptual difficulty, degree of required abstraction, complexity of detail, and difficulty of linguistic expression. Initially , five degrees of difficulty will be associated with each dimension. Each problem occupies a single cell in its cube, accompanied by a correct and several incorrect answers, and by supple- mentary inquiries associated with these, and, as conclusions about the user’s performance, a set of four direction indicators suggesting to the controller a location for the next problem to be posed; there may also be specific messages to be relayed to the user, e.g., “Are you confusing sine with cosine?”

The first-time student user, having chosen the topic-cube, will normally be asked one of the problems in its center, and proceed to more or to less difficult problems in the various dimensions. This is according to the cube controller’s assignments, which take into account the suggestions from the individual cell together with a consideration of the trajectory so far followed, and of course the availabiIity of fresh problems in the direction chosen. Repeated loops and other irrational trajectories are avoided by this kind of design. The result of a session is not only a position judged to be the student’s proper present location in the cube, but also a record of the path

SCIENTIFIC PROBLEM-SOLVING ABILITIES

b I

CHAIN OF

THOUGHT

ANALYZER

ARCHIVES

b

27 1

CHAIN OF

THOUGHT

ANALYZER

ARCHIVES

\

CONT R O U E R PROCESSOR /

STUDENT

USER

COMMUNICATOR

/ SPECIALIST

INTERFACE

Fig. 2. DIPS design showing major components within zones.

followed and the messages so far emitted. Together, all of this forms the material for a preliminary diagnosis which can then be communicated to the user. Notice in Figure 3 that the dictionary is located within the student user communicator. As an example of the size of the dictionary needed for a science content area such as mechanics in physics, DIPS staff found that a mechanics chapter in a high school physics text contained about 5000 words, including about 800 unique words. This suggests that about 1000 words are required for a “module” such as mechanics in physics.

The student user who has already accumulated a history of interaction with the system will be led by the controller through the problems in a more discriminating manner, taking advantage of previous records. The announced diagnoses, also, can then contain an element of comparison. The task of issuing the diagnoses is performed within the communicator zone, on the basis of the indications furnished by the diagnoser proper.

Initially, the cells in the cubes will be stocked by the physics experts, using a standard question-and-answer modular form. The initial versions of the cubes will be tried out and calibrated on student novices, and information from the earliest archives will be used to help in the construction of further problems.

The archives zone of the DIPS system is of major importance, contributing enor- mously to its usefulness and also to its vitality. It is there that a record is kept of each session of every student user and this has three destinations: (1) for use as mentioned above in future sessions with the same student, (2) for the information of the teacher (only with the student’s permission), and (3) for the use of researchers and for the guidance of the system designers.


rl CONTROLLER

Studen2 l - - I STUDENT

INTERFACE

/

CHAIN OF

THOUGHT

ANALYZER

/ rf O‘JERALL

CONTROLLER

I I

SPECIALIST

INTERFACE

Specialist r$;l

ONT ROLLE I; E Fig. 3 . DIPS design showing all major components.

The archives are certain to provide a very rich and varied body of information on the learning process, including its difficulties. What are the questions most often missed? What are the favorite wrong answers? Which expected wrong answers occur? What are common sequences of responses? How do misconceptions develop? How do responses vary from class-to-class, school-to-school, etc. It is impossible to more than hint at the wealth of information which can be made available, because of the limited space in this paper. Clearly, the potential for future science education research is very large.

Implications for Science Teaching, Learning, and Future Research

A question often put to DIPS project workers is, “Will your system replace “real” teachers?” Our answer to this question has been and will continue to be no! DIPS is intended as a unique tool for teachers and students to use in their attempts to raise levels of science problem solving through more accurate diagnoses of problem- solving states. Diagnosis is only one of many facets of teaching that can result in a deeper understanding of science and, thus, a greater ability to solve problems within a field of science. Once an accurate diagnosis of a student’s problem-solving state is available, what then? How should the student work to improve that state? The answer to this question depends on many things, including assumptions about the nature of science, purposes of education, etc., and science education researchers continue to


seek answers to this very basic question. The role of “intelligent” computer systems in science education of the future is unclear now, but “surely they will become increasingly important. The existence of such machines suggests a greater likelihood of learning with less dependence on human teachers, at least for certain aspects of teaching as it is now practiced. We recognize there are potential dangers to an education system that relies too heavily on machines for decision making; the same is true for a medical system or other traditionally people-intensive systems. This is a legitimate concern that will require vigilance by those who assess the effects of changes in the system.

Future science education research will be shaped, to a considerable degree, by the nature of science education as it exists in schools and elsewhere. If machine systems such as DIPS become an integral part of science education, there will be new roles for science education researchers that are now barely perceptible. Classroom interaction analysis, for example, would have to include student interaction with intelligent machine tutors as well as with human teachers; Card, Moran, and Newell (1983), Thomas (1984), and Monk (1985) provide information related to this issue. Research on factors which facilitate student science learning could not ignore machines if students began to spend significant amounts of time interacting with them. The role of the textbook could change considerably from its central position in most of today’s classrooms. If, at some point in the future, intelligent machines are commonly used in science edcation, it seems likely that much of what we now see as mainstream R&D activity, including teacher education, will be changed accordingly. Interdisci- plinary efforts, such as DIPS, will likely include science education specialists as well as computer scientists, science content specialists, linguists, etc.

Human science teachers and science education researchers will not be replaced by the “intrusion” of expert systems into education, but surely their roles will change. There were no dramatic changes in science education in the year 1984; perhaps 2001 will be different.

Acknowledgment

The authors wish to thank the many people who have contributed ideas to the DIPS project. In particular, Sue Szabo for her long-term commitment and valuable ideas on system architecture, Marlyn Dawson for her continuing work, especially with the archives system and user- friendly interface, Zofia Roberts, Kevin Rappoport, Jim Groh, and Lany Hall. Support from FSU Graduate Research Studies Dean, Robert Johnson, has been helpful and is much appre- ciated. A special thanks to FSU Computer Science Head, Abe Kandel, for his early leadership and encouragement. Suggestions from a number of Institute for Cognitive Sciences members (Bob Gagne and George Weaver, in particular) have been very helpful.

References

Aguirre, J., & Erickson, G . (1984). Student’s conceptions about the vector characteristics of three physics concepts. Journal of Research in Science Teaching, 21, 438-457.

Anzai, Y., & Yokoyama, T. (1984). Internal models in physics problem solving. Cognition and Instruction, 4, 397-450.

Buchanan, B., & Feigenbaum, E. (1978). DENDRAL and Meta-DENDRAL: Their applications dimension. ArtiJicial Intelligence, 11, 5-24.


Card, S . , Moran, T., & Newell, A. (1983). The psychology of human-computer interaction. Hillsdale, NJ: Erlbaum.

Case, R. (1980). Intellectual development and instruction: A neo-Piagetian view. In A. Lawson (Ed.), The psychology of teaching for thinking and creativity. 1980 AETS yearbook, Columbus, OH: ERIC.

Champagne, A., Klopfer, L., & Anderson? J. (1980). Factors influencing the learning of classical mechanics. American Journal of Physics, 48, 1074-1079.

Chi, M . , Feltovich, P., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5, 121-152.

Clement, J. (1979). Mapping a student’s causal conceptions from a problem- solving protocol. In J. Lockhead & J. Clement (Eds.), Cognitive Process Instruction. Philadelphia: Franklin Institute.

Clement, J. (1982). Student’s preconceptions in introductory mechanics. Amer- ican Journal of Physics, 50, 66-7 1.

diSessa, A. (1 982). Unlearning Aristotelian physics: A study of knowledge-based learning. Cognitive Science, 6, 37-75.

Duda, R., Gaschnig, J., & Hart, P. (1979). Model design in the PROSPECTOR consultant system for mineral exploration. In D. Michie (Ed.), Expert systems in the- micro-electronic age. Edinburgh: Edinburgh University Press.

Ericsson, K., & Simon, H. (1984). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.

Erman, L., Hayes-Roth, F., Lesser, V. , & Reddy, D. (1980). The HEARSAY- I1 speech-understanding system: Integrating knowledge to resolve uncertainty. Com- puting Surveys, 12, 213-253.

Gagne, R. (1966). Human problem solving: Internal and external events. In B. Kleinmutz (Ed.), Problem solving: Research, method, and theory. New York: Wiley.

Greeno, J. (1976). Indefinite goals in well-structured problems. Psychological Review, 83, 479-49 1.

Gunstone, R., & White, R. E., (1981). Understanding of gravity. Science Edu- cation, 65, 291-299.

Hayes-Roth, F., Waterman, D., & Lenat, D. (1983). Building expert systems. Reading, MA: Addison-Wesley .

Larkin, J., McDermott, L., Simon, D., & Simon, H. (1980). Expert and novice performance in solving physics problems. Science, 208, 1335-1342.

Larkin, J. (1981). Cognition of learning physics. American Journal of Physics,

Lugar, G. (1981). Mathematical model building in the solution of mechanics problems: Human protocols and the Mecho trace. Cognitive Science, 5, 55-77.

McCloskey, M. Caramazza, A., & Green, B. (1980). Curvilinear motion in the absence of external forces: Naive beliefs about the motion of objects. Science, 210,

McDermott, L. (1984). Research on conceptual understanding in mechanics. Physics

Monk, A. (Ed.) (1985). Fundamentals of human computer interaction. New

Newell, A., & Simon, H. (1972). Human problem solving. Englewood Cliffs,

Pople, H. (1977). The formation of composite hypotheses in diagnostic problem

49, 534-541.

1139-1141.

Today, 37 (7), 24-32.

Y ork: Academic.

NJ: Prentice-Hall.

solving: An exercise in synthetic reasoning. ZJCAI, 5, 1030-1037.


Preece, P. (1976). The concepts of electromagnetism: A study of the internal representation of external structures. Journal of Research in Science Teaching, 13,

Reif, F. (1983). Understanding and teaching problem solving in physics. Lectures at the International Summer School on Physics Education. LaLonde Les Maures, France, June-July .

Reif, F., & Heller, J. (1982). Knowledge structure and problem solving in physics. Educational Psychologist, 17, 102-127.

Shortliffe, E. (1976). Computer-based medical consultation: MYCIN. New York: American Elsevier.

Simon, D., & Simon, H. (1978). Individual differences in solving physics problems. In R. Siegler (Ed.), Children’s thinking: What develops? Hillsdale, NJ: Erl- baum.

Sleeman, D., & Brown, J. (1982). Intelligent tutoring systems. New York: Aca- demic.

Smith, M., & Good, R. (1984). Problem solving and classical genetics: Suc- cessful vs. unsuccessful performance. Journal of Research in Science Teaching, 21,

Stewart, J. (1982). Two aspects of meaningful problem solving in science. Sci- ence Education, 66, 731-749.

Stewart, J., & Atkin, J. (1982). Information processing psychology: A promising paradigm for research in science teaching. Journal of Research in Science Teaching,

Thomas, J. (Ed.) (1984). Human factors in computer systems. Norwood, NJ: Ablex.

Trowbridge, D., & McDermott, L. (1980). Investigation of student understanding of the concept of velocity in one dimension. American Journal of Physics, 48,1020-1028.

Trowbridge, D., & McDermott, L. (1981). Investigation of student understanding of the concept of acceleration in one dimension. American Journal of Physics, 49, 242-25 3.

Weiss, & Kulikowsky (1979). EXPERT: A system for developing consultation models. IJCAI, 6 , 942-947.

White, B. (1983). Sources of difficulty in understanding Newtonian dynamics. Cognitive Science, 7 , 41-65.

5 1 7-524.

895-912.

19, 321-332.

Manuscript accepted October 25, 1985

Documents

Human and machine diagnosis of scientific problem-solving abilities