MedicalEducation 1988,~~,279-286

Problem-solving skills, solving problems and problem-based learning

G. R. NORMAN

Programme f o r Educational Development, McMaster University, Hamilton, Ontario

Summary. This paper reviews the empirical evidence in support of the three concepts in the title. To the extent that a skill should be a general strategy, applicable in a variety of situations, and independent of the specific knowledge of the situation, there is little evi- dence that problem-solving skills, as described and measured in medical education, possess these characteristics. Instead there is an accu- mulation of evidence that expert problem- solving in medicine is dependent on ( I ) a wealth of prior specific experiences which can be used in routine solution of problems by pattern recognition processes, and (2) elabo- rated conceptual knowledge applicable to the occasional problematic situation.

The use of problem-based learning (PBL) as an educational strategy is explored. In particu- lar, the evidence suggesting the compatibility of PBL with this theory of expertise is discus- sed. Finally, I review some issues in the design of PBL curricula from the perspective of the proposed model of expertise.

Key words: *problem solving; *education, medical; curriculum; clinical competence

Introduction

T w o catchphrases have crept into the lexicon of medical educators over the past two decades- problem-solving skills and problem-based learn- ing. Often the words are used interchangeably,

Correspondence: Dr Geoffrey R. Norman, De- partment of Clinical Epidemiology and Biostatistics, Room 2C14, Health Sciences Centre, McMaster University, Hamilton, Ontario, Canada L9H 2K9.

or it is assumed that problem-based learning (PBL) is a means to the end of acquiring problem-solving skills (PSS). I propose that the two words should have very different meanings and that PBL as an instructional strategy is unrelated to the learning of PSS. Further, although it seems an obvious and natural assumption that problem-solving skills are necessary for solving problems, I will propose a distinction between the two terms, and indicate that the majority ofproblems in clinical medicine are solved through mental strategies that do not fit into the conventional definition of ‘problem- solving skills’. Finally, I suggest that the basic assumptions of PSS are open to question, whereas the use of PBL as a method of instruction is firmly grounded in both theory and evidence.

Assumptions of PSS

The assumption underlying the idea that in- struction should be directed to an acquisition of PSS is that such skills, once acquired, are a prerequisite for adaptation and survival. Once individuals have acquired PSS, i.e. ‘become a successful problem-solver’, they can go forth and tackle problems of all sorts, from carpentry to cardiology, confident that they have the mental faculties to arrive at a successful conclu- sion. If they lack knowledge to solve the problem, no difficulties arise since they have the self-directed learning skills, acquired in the practice of problem-solving skills, to go and find the knowledge. As one sage said ‘Give a man a fish and he eats for a day-teach a man to fish and he eats for a lifetime’. An appealing

279

280 G. R. Norman

idea. Unfortunately, if the only body of water nearby is a polluted stream, a ready source of cash to dispense at the local fish and chip shop may go further in aiding survival than all the fishing skills in the world. Similarly, the under- lying assumptions of this model of problem- solving do not square very well with the evidence from empirical studies.

The notion of a general problem-solving process, often described in other terms such as ‘clinical methods’, ‘clinical judgement’ or ‘cli- nical reasoning skills’, has been subject to several investigations dating back over a decade (Elstein et al . 1978; Barrows ef al . 1981; Neufeld et al . 1981; Gale & Marsden 1983) and some recent review articles (Groen & Patel 1985; Barrows & Feltovich 1987). The results of these studies provide some evidence of a general mental strategy, a process which has been labelled ‘the hypothetico-deductive method’ (Elstein et al. 1978). The clinician generates several diagnostic hypotheses early in the clinical encounter, and proceeds to gather data from history, physical examination and the laboratory to support or refute these hypotheses. Much has been written about this latter process of gathering data to accept or refute hypotheses, both in textbooks of clinical diagnosis and in philosophical and empirical writings in medical education. Certainly, this appears at the heart of the ‘clinical reasoning process’, a process which ‘represents conver- gent, vertical or deductive reasoning that in- stantiates to the particulars of the patient from a set of generic explanations’ (Barrows & Felto- vich 1987).

Unfortunately, the process, however ele- gantly described, has some very un-skill-like characteristics. Students, even in the first year of medical school, use exactly the same pro- cess, so if it is a skill, it is not learned in medical school (Neufeld et al . 1981). The main difference between expert clinicians and stu- dents is that experts generate better hypotheses-not a characteristic of the skill itself. Further, any attempt to identify variables which are predictive of successful problem- solving has also shown that the correctness of the specific hypotheses, rather than any process variable, was the strongest predictor of success (Barrows ef a l . 1981).

Furthermore, evidence is accumulating that problem-solving skills, as measured by a num- ber of variables obtained from work-ups of simulated patients and written patient manage- ment problems (data gathered, diagnostic accuracy, appropriateness of management, etc.) typically correlate at about the 0’2 level across problems (Elstein et al . 1978; Norcini et a / . 1983; Norman et a l . 1985b). If a skill is sufficiently general to be applied in a variety of situations, then problem-solving, measured in these studies, does not appear to have this characteristic.

Skills have some other characteristics. For example, if problem-solving were a general skill, we would expect that it would be re- latively independent of knowledge. Careful studies by the American Board of Emergency Medicine (Maatsch et a l . 1982) have shown the converse; if you present candidates with enough problems to get a reliable estimate of their ability to solve problems, the resulting score correlates 0.92 with a separate measure of knowledge assessed by multiple choice ques- tions, so that problem-solving as measured in these studies is heavily influenced by know- ledge. These results stand in direct opposition to early results from studies of patient manage- ment problems (McGuire et d. 1972), where a low correlation with multiple choice tests was used to support the assertion that patient man- agement problems (PMPs) measured problem- solving skills. It now appears that the low correlation was due to low correlation of the PMP scores across cases which placed an upper limit on the possible correlation with other measures, rather than any independence from knowledge.

Lest these results are viewed as a perversion of medical education, they square remarkably well with studies in fields ranging from chess to artificial intelligence (AI). In the former category, studies of expertise (Chi et a l . 1987) have repeatedly demonstrated that the expert is distinguished, not by the possession of any general skills, but by the ready availability from memory of appropriate knowledge to resolve the problem. The expert is an expert primarily because he has seen it all before.

A similar conclusion arises from the A1 literature (Waldrop 1984). Two decades ago,

Problem-solving and problem-based learning 28 I

the dream was to invent machines that could take on any problem. One of the first program- mes was called the general problem solver (GPS) (Newell & Simon 1972). Unfortunately, it did not work too well. It was capable of solving a dozen or so problems of the type that psychologists take perverse pleasure in inflict- ing on first-year psychology students. Yet it was incapable of dealing with the complex problems encountered by real-world experts. Modern A1 programmes have much more modest goals-they attempt to succeed in well- circumscribed domains and they do this by programming endless series of ‘If-then’ rules, i.e. knowledge. In the final analysis, the very best do a bit better than an average participant in the game (chess, computer programming, or cardiology), but none can beat the real expert.

General problem-solving skills may exist. However, they do not appear to be assessed by any of the measures of problem-solving we have used in medical education. Furthermore, studies in a variety of disciplines suggest that, even if these skills do exist, they do not go far in explaining the acquisition of expertise.

Problem-solving skills versus solving problems

Despite the fact that expert clinicians, and experts in many other disciplines, do not appear to be excessively endowed with problem-solving skills, they are clearly success- ful at solving problems with a considerable degree of efficiency and accuracy. But if the problem is solved, how can I claim that this activity does not constitute ‘problem-solving’? To challenge what appears, at face value, to be a truism, one must examine some basic defini- tions of problem-solving. Davis (1973) prop- oses a definition of a problem as ‘a stimulus situation for which an organism does not have a ready response’. This definition implies that problems, like beauty, are in the eye of the beholder. The fact that a doctor has solved a problem of angina does not, of itself, provide any indication that the doctor has engaged in ‘problem-solving’, any more than the observa- tion that most adults can solve the problem of multiplying 8 times 9 suggests that this solu-

tion requires application of problem-solving skills.

Davis (1973) supports this view in the follow- ing discussion:

‘It follows from our definition that a “problem” for a naive organism may not be a “problem” at all for a more sophisti- cated organism . . . Strictly speaking, our definition further implies that once we solve a given problem, we should call it something else’ (p. 13).

Indeed, an informal study I have conducted over several years with clinician friends from a variety of disciplines indicated that they were intellectually challenged (i.e. engaged in problem-solving) by about one patient in 20, although many individual estimates were closer to I in 100.

How does the clinician arrive at an appropri- ate conclusion on the other 95 occasions? It appears that in large part the process leading to a correct diagnosis results from a comparison of the current situation with previous instances in memory. Although the rules of medicine, expressed in the basic science concepts or the textbook descriptions of diseases, provide a framework for the processing of information from patients, expert diagnosis rarely invokes these concepts (Pate1 et a l . 1986). We know that in fields which depend on visual recognition, like radiology and dermatology, correct identi- fication of lesions occurs very rapidly, of the order of 10 seconds, and errors are associated with longer times (Norman et al . 1985a). This finding suggests a rapid process, and does not allow much time for the detection, analysis and integration of specific features.

Of course, it can be argued that the process is one of hypothesis generation and testing, ex- cept that the expert is simply unaware of his mental processes (Barrows & Feltovich 1987). There is ample evidence that verbal reports are fallible sources of evidence of mental processes (Ericsson & Simon 1980), although any claim of a particular form of unaware or subcon- scious process must be based on confirming evidence rather than conjecture.

Direct evidence of a mechanism involving comparison to prior instances in memory is provided by a study we recently conducted. In this study, a sample of general practitioners

282 G. R. Norman

was given a single exposure, lasting a few seconds, to a photograph of a skin lesion. Four to six weeks later, they were given a diagnosis test containing both these old examples and new examples. The single prior exposure re- sulted in an increase in accuracy of 10% and a decrease in response time of 5 seconds, com- pared to a set of new slides. This phenomenon has often acquired the pejorative label of ‘just pattern recognition’, presumably because it appears to the expert to occur so rapidly and effortlessly. Such a label belies the complexity of the mental processes, involving the rapid retrieval of an appropriate match based on a limited number of salient cues, which give rise to the process.

Generalizations from highly visual domains such as dermatology or radiology may not be warranted. However, the evidence from the early studies of clinical reasoning, using gener- alist doctors and clinical cases, is consistent with this view. In particular, one analysis showed that if doctors considered the correct diagnosis in the first 5 minutes of the encoun- ter, they had a 95% chance of arriving at the correct diagnosis; conversely, if they failed to consider the correct diagnosis in the first 5 minutes, they had a 95% chance of not reaching the diagnosis. Of course, the time scale in these studies is far longer than the few seconds associated with visual diagnosis. However, times measured in seconds are clearly in- appropriate for the processes of data gathering in the actual patient encounter. What is evident is that in both settings, the diagnostic process is coniplete long before all the data are gathered or analysed, and bears little resemblance to the dispassionate, objective, rational information- gathering of the mythical scientific clinician (Cutler 1979).

O f course, the process leading to the gencra- tion of a diagnostic hypothesis is then followed by a longer period of data-gathering, weighting of features, and, very occasionally, a change in decisions. However, this ‘clinical reasoning process’, which appears to have all the charac- teristics of logical Objective inquiry, is, in fact, prone to self-fulfilling prophesies, as a result of such biases as ‘confirmation bias’ (Lord et a l . 1979) and ‘hindsight bias’ (Fischoff 1975),. Cer- tainly one of the early studies in medicine

demonstrated clearly that the process is largely a search for data to confirm prior hypotheses (Barrows et al . 1981).

Problem-based learning What then of problem-based learning? If the game is not to teach the problem-solving process, how does one justify the use of clinical problems as the central feature o f a curriculum? The answer is straightforward. PBL is simply a case of learning ‘stuff as students work their way through a clinical problem. In general, the ‘stuff is unspecified. Some of it is the usual stuff of medicine-Krebs cycles and Starling Laws. However, the problem is unbounded, and the stuff also encompasses epidemiology, psychology, pharmacology, and just about any other -0logy available in medical, behavioural or social science. In short, instead of learning a self-contained and quickly forgotten body of facts called physiology, the information is gra- dually assembled in a helter-skelter way as the student reflects on the clinical problems. Each student and each group may choose to reflect on different issues in a problem.

Note that it is not necessary to ‘solve’ the problem, i.e. to ‘acquire problem-solving skills’, in order to engage in PBL. You can enter anywhere, leave anywhere, and leave the poor paper patient in a state of suspended animation or inevitable demise if you choose. The primary objective of problem-based learn- ing is to accumulate the concepts of medicine in the context of a clinical problem.

O f course, there may be some secondary reasons for discouraging the demise of paper patients. One is motivational-students enjoy the opportunity to practise the doctoring game. After all, that is presumably the reason they applied to medical school. There is a second reason related to the pattern recognition pro- cess described earlier. T o the extent that exper- tise in medicine is contingent on a large store- house of prior examples, then the work-up of problems in a tutorial setting is a beginning to the accumulation of these examples.

What could be the possible advantages of this curricular chaos? There are two advantages which clearly emerge from the literature. The first is that students clearly prefer this approach. There is very little evidence as yet to

Problem-solving and problem-based learning 283

justify any claim that they emerge very diffe- rent from other students (Schmidt et al . 1987), but they certainly enjoy themselves far more along the way. The lack of any large demon- strated difference may reflect an inadequacy in our choice of measures. However, it is also apparent that problem-based learning is no more expensive than other curriculum options. We find ourselves presented with two radically different approaches, which cost the same, and have the same product, but one is more enjoy- able than the other. The choice appears analo- gous to the choice between sex and artificial insemination.

Nevertheless, there is a second advantage which is rooted in psychological theory, and may provide a strong rationale for the approach. It is well established that knowledge is much better remembered or recalled in the context in which it was originally learned. An everyday example is that, upon meeting a colleague or student in the grocery store, an unfamiliar context, you may have difficulty recalling the person’s name. If you meet the same person at work on Monday morning, there will be no difficulty whatsoever. One of the first experimental demonstrations of the phenomenon (Godden & Baddeley 1975) in- volved Royal Navy divers who memorized strings of 20 words, some on land and some underwater. The words learned on land were better recalled on land, and the words learned underwater were better recalled underwater.

The task of clinicians is to apply all their learned knowledge and skills to the solution of a patient’s problem. Thus, it makes sense to learn the prerequisite knowledge in the same context-in the context of a patient problem. There is some evidence, as yet preliminary, that indeed this hypothesis is borne out. A comparison of a PBL school, Maastrict, to a conventional school in the Netherlands (Claes- sen & Boshuisen 1985), showed that students in the PBL curriculum were indeed better able to recall information in the context of patient problems. However, much more work needs to be done.

Implications for education If one accepts these arguments, there are several implications for the use of problem materials in

problem-based learning. It is unlikely that the process of working through the problem adds to any repertoire of general problem-solving skills. Since what is learned is not a general strategy, independent of the problem content, problems are far from interchangeable. Careful thought must be given to the facts determining the choice of problems to be used in problem- based curricula.

A body of research has evolved over the past few years in response to the need of problem- based curricula for appropriate selection criteria for problems. Some of this work has origins in the classic paper of Kerr White (1961) in which he argues that the setting of most medical education, in the tertiary care hospital, is in- appropriate to meet the health needs of society. From this concept, some authors have argued that curricula should be dictated by the priority health problems of the community served by the health care system. Empirical studies (Neufeld & Chong 1984) have developed tech- niques to arrive at a definition of priority problems based on considerations of treatabil- ity, seriousness, prevalence and ‘prototype value’.

Although the argument that students should graduate equipped to manage the problems that they encounter is unassailable to all but the most ardent academics, there is nevertheless some reason to separate pedagogical factors for selection of problems for learning from the epidemiological factors determining the ter- minal objectives of the curriculum. To the extent that the problems provide a context for the learning and recall of concepts, there may be advantages to using problems which are more ‘prototypical’ for instruction in the precli- nical years (Bordage & Zacks 1984). As one example, ulcerative colitis may be a more pedagogically effective means to understand the mechanisms of gut inflammation than infant gastroenteritis which is far more prevalent and treatable.

A second issue is the degree of reality or ‘fidelity’ which may be appropriate for learn- ing. It would appear that the closer a problem approximates real life, the merrier, as has been argued by some authors. For example, Bar- rows & Feltovich (1987) assert, without proof, that ‘Problems used for . . . teaching and eva-

284 G. R. Norman

luation of clinical reasoning in medical educa- tion will be most effective if they present the problem-solver with ill-structured problems’

On closer inspection the issues may be far more complex. Greater fidelity has several potential advantages. Problems with more fidelity, like live simulated patients, may be more likely to transfer to the ‘real world’ of clinical medicine. Further, these problems may be more motivational for students. Neverthe- less, such problems carry enormous contextual information, ranging from features of personal and family history to the kind of clothes the person is wearing and the colour of his hair. Such contextual clues will inevitably aid subse- quent recall. However, to the extent that these features are unrelated, or spuriously related to the patient problem, they may be a handicap rather than an asset to learning. Certainly, the student’s task now involves the additional com- plexity of sorting out signal from noise, a problem which is not necessary if more proto- typical problems are used.

The resolution of the issue may be an explicit consideration of educational level as an indepe- nent variable; beginning with problems which are more prototypical and devoid of extraneous contextual information, and moving to more complex, real-world, and higher fidelity prob- lems over time.

Should we present a long series of cases of a particular condition, or should problems be presented in a ‘mixed-series’ with problems of other kinds? Frequently, the approach is the former. Students discuss each condition in turn, focusing on, for example, the signs and symptoms of myocardial infarction. Textbooks are written describing one disease at a time. The difficulty with this approach is that a consistent bias of human information proces- sing is the tendency to seek confirming in- formation and ignore disconfirming cues. The bias has been discussed at length in the philo- sophical literature by Popper (Popper 1959) and empirically demonstrated on a number of occa- sions (Lord et a l . 1979). The real challenge facing the clinician is to distinguish among possible causes (e.g. seeking the features that distinguish between myocardial infarction and acute pericarditis) rather than ruling in or out

(P. 90).

either condition. In these circumstances, it would seem that students should rapidly prog- ress to dealing with problems in a ‘mixed series’, contrasting likely diagnoses to discover discriminating features, after achieving a basic level of understanding of each condition.

Implications for evaluation

The concepts I have discussed have clear im- plications for evaluation. First, it is now evi- dent why efforts to assess ‘problem-solving skills’ over the past decade have led to many studies showing small effects and few outstand- ing successes. Two features contribute to the frequent finding of small differences between experts and students.

(I) The lack of generalizability of scores across different content domains suggests that most studies have used too small a sample of cases to ensure a stable estimate of perform- ance.

(2) Even if a large sample of cases were sampled, the focus on measures of data- gathering, which occurs after the critical event of hypothesis generation and is vulnerable to confirmatory biases, ensures that critical differ- ences in performance will be missed.

The empirical demonstration of differences between experts and students in tasks related to perception and memory suggests a new area for exploration in measuring student performance. Indeed, several studies using measures of perception (of visual materials) and memory (of numerical laboratory data) have demonstrated their ability to detect large, consistent differ- ences between individuals even after short testing periods (Norman 1987). For example, asking subjects at two levels, final-year medical students and expert clinicians, to recall numer- ical laboratory data from two protocols, in- volving a total testing time of about 1 5 mi- nutes, showed reliability coefficients within groups exceeding 0.8. Similar reliabilities were reported on a measure of accuracy based on brief exposure to IOO skin lesions.

Does the emphasis on the role of organized knowledge, as opposed to acquisition of gene- ral skills, signal a return to traditional (i.e. multiple choice question) testing methods? Hopefully not. It should be clear from this

Problem-solving and problem-based learning 285

discussion that the domains of competence I have described are not conventionally tested by such methods; indeed the fact that these struc- tures appear to evolve primarily after formal education is complete strongly suggests that, whatever the nature of this ability, it is not being assessed by conventional methods.

Conclusion

Many developments in medical education over the past two decades have focused on attempts to instil problem-solving or clinical reasoning skills which may be applied to a diversity of problem situations. However, it is now appa- rent that the search for such skills was quixotic. There is a convergence of evidence from a variety of fields of inquiry that expertise is characterized, not by the possession of any superior general strategies, but by the availabil- ity of an extensive organized body of special- ized knowledge. Further, recent research has emphasized that the characterization of expert knowledge simply as organized networks of scientific rules and concepts ignores the central role of experiential knowledge in expert per- formance.

Nevertheless, although this re-con- ceptualization of expertise results in a de- emphasis of problem-solving strategies, there is some indication that the theory is highly com- patible with problem-based learning as a curri- culum strategy in medical education.

Acknowledgement

This research was supported by the Natural Sciences and Engineering Research Council of Canada.

References

Barrows H.S., Neufeld V.R., Feightner J.W. & Norman G.R. (1978) A n Analysis of the Clinical Methods of Medical Students and Physicians. Final report to the Ontario Ministry of Health, Hamil- ton, Ontario.

Barrows H.S. & Feltovich P.J. (1987) The clinical reasoning process. Medical Education 21, 8-1.

Barrows H.S., Norman G.R., Neufeld V.R. & Feightner J.W. (1981) The clinical reasoning pro- cess of randomly selected physicians in general

medical practice. Clinical and Investigative Medicine - 5. 4P-5s.

Bordage G. & Zacks R. (1984) The structure of L .~ .,

medical knowledge in the memories of medical students and general practitioners: categories and prototypes. Medical Education IS, 40616.

Chi M., Glaser R. & Farr M. (Eds) (1987) The Nature of Expertise. University of Pittsburgh Press, Pitts- burgh.

Claessen H.F.A. & Boshuisen H.P.A. (1985) Recall of medical information by medical students and doctors. Medical Education 19, 61-7.

Cutler P. (1979) Problem-Solving in Clinical Medicine: From Data to Diagnosis. Williams and Wilkins, Baltimore.

Elstein A.S., Shulman L.S. & Sprafia S.A. (1978) Medical Problem-Solving: A n Analysis of Clinical Reasoning. Harvard University Press, Cambridge, USA.

Davis G. A. (1973) Psychology of Problem-Solving: Theory and Practice. Basic Books, New York.

Ericsson K.A. & Simon H.A. (1980) Verbal reports as data. Psychological Review 87. 215-52.

Fischoff B. (1975) Hindsight=foresight: the effect of outcome knowledge on judgement under uncer- tainty. journal of Experimental Psychology: Human Perception and Performance I, 288-99.

Gale J . & Marsden P. (1983) Medical Diagnosisfrom the Student to the Clinician. Oxford University Press, Oxford.

Godden D.R. 81 Baddeley A.D. (1975) Context- dependent memory in two natural environments: on land and underwater. British Journal of Psychol- ogy 66, 325-32.

Groen G. & Patel V.L. (1985) Medical problem- solving: some questionable assumptions. Medical Education 19, 95-100.

Lord C., Ross L. & Lepper M.R. (1979) Biased assimilation and attitude polarization: the effects of prior theories on subsequently considered evi- dence.Journal ofPersonality and Social Psychology 37, 2098-1 10.

Maatsch J.L., Munger B.S. 81 Podgorny G. (1982) Reliability and validity of the Board examination in emergency medicine. In: Emergency Medicine Annual, I. Appleton, Century, Croft, Norwalk.

McGuire C.H., Solomon L.M. & Bashook P.G. (1972) Handbook of Written Simulations: Their Con- struction and Analysis. Center for Educational De- velopment, Chicago.

Neufeld V.R. & Chong J.P. (1984) Problem-based professional education in medicine. In: Education

fo r the Professions: Quis Custodit? (ed. by S. Good- lad). Society for Research in Higher Education, Nelson.

Neufeld V.R., Norman G.R., Barrows H.S. & Feightner J. W. (198 I) Clinical problem-solving of medical students: a longitudinal and cross- sectional analysis. Medical Education 15, 2632.

Newell A. & Simon H.A. (1972) Human Problem- Solving. Prentice Hall, Englewood Cliffs.

Norcini J.J., Swanson D.B., Gross L.J. & Webster

286 G. R.

G.D. (1983) A comparison of several methods for scoring patient management problems. Proceedings of the zznd Conference of Research in Medical Educa- tion. Washington, DC.

Norman G.R. (1987) Measurement characteristics of cognitive probes. Proceedings of the Twenty-sixth Conference on Research in Medical Education, Washing- ton DC.

Norman G.R., Muzzin L.J. & Rosenthal D.R. (Iy8Sa) Expert novice differences in perception and cate- gorization in dermatology. Presented at the Amer- ican Educational Research Association annual meeting, Chicago.

Norman G.R., Tugwell P., Feightner J.W., Muzzin L.J. & Jacoby L.L. (Iy8Sb) Knowledge and clinical problem-solving. Medical Education 19. 344-56.

Patel V . L . , Arocha J. & Groen G. (1986) Strategy

Norman

selection and degree of expertise in medical reason- ing. Proceedings of the Cognitive Science Society, 78-1.

Popper K.A. (1959). The Logic of Scientijc Discovery. Basic Books, New York.

Schmidt H.G., Dauphinee W.D. & Patel V . L . (1987) Comparing the effects of problem-based and conventional curricula in an international sample. journal of Medical Education 62, 305-15.

Waldrop M.M. (1984) The necessity of knowledge. Science 223, 1 2 7 ~ 8 2 .

White K.L. (1961) The ecology of medical care. New EnglandJournal of Medicine 265, 885-9.

Received 14 December 1987; acceptedfor publication IsJanuary 1988