Medical Education, 1977, 1 1 , 329-334

Assessment of problem-solving ability

J. M A R S H A L L

Royal Australian College of General Practitioners, North Adelaide, South Australia

Key words: *PROBLEM SOLVING; *EDUCATIONAL MEASUREMENT ; *EDUCATION, MEDICAL ; CLINICAL SKILLS ; DIAGNOSIS; AUSTRALIA

Assessment of problem-solving ability

The assessment of clinical competence in medicine has received considerable attention in the past few years, with general agreement that the variety of skills to be measured demands a variety of assess- ment techniques to be used. Newble (1975) has recently reported the current position. It is therefore seen to be important when assessing the competence of practising family doctors to gain a t least a measure of:

(1) recall of factual knowledge; (2) interpretation; (3) affective behaviour; (4) psychomotor skills; (5 ) problem solving ability. (1) Recall of factual knowledge is easily measured

by multiple choice questions, and in the past these have been the basis of most assessments at primary, secondary, tertiary and postgraduate examinations.

(2) Interpretation tests a slightly higher level of learning in that an application is required of the knowledge recalled. Again this may be adequately tested by many methods, including multiple choice questions particularly of the more complex type.

(3) The assessment of affective behaviour has until recently attracted little attention. This be- havioural assessment is seldom catered for in the academic establishments in medicine, which are geared to imparting as many facts as possible, with patients usually being regarded as disease processes rather than individuals. The measurement of affec- tive behaviour has been developed using simulated

Correspondence: Dr J. Marshall, Family Medicine Programme, Royal Australian College of General Prac- titioners, 183 Tynte Street, North Adelaide, South Australia 5006.

interview techniques or in group conference situations, and may be assessed fairly accurately using structured rating forms. (4) Psychomotor skills are easily assessed by

experienced persons observing the use of hands and instruments. Video-tape facilities can be used both as a check on the accuracy of observations and as a method providing information on performance to the individual being assessed. This method is also useful in the measurement of affective behaviour.

(5) The most diffcult area, and perhaps the most neglected has been the assessment of problem solving ability. Within the Royal Australian College of General Practitioners assessment of this ability has been through the use of patient management problems (PMPs) in both medical and surgical areas.

The account which follows briefly describes the format of the tests used and the various stages in the development of the structure and scoring procedures which appeared necessary as the PMPs were used over a period of time. The design of the tests has followed that initially used by McGuire & Babbott (1967) in Chicago and introduced to the College examination by Wesley Fabb (1967) who has contributed valuable and pioneering work in the whole area of clinical evaluation.

The Patient Management Problem is produced as a booklet in which the information required to solve the problem is concealed in the form of invisible ink. Development of this is necessary before any item of information may be obtained, allowing responses to be made-as would occur in the real life situation. The ink was developed by applying a dilute bleach solution with a cotton applicator. Since 1976 the information has been hidden by colour overprint which reduces the cost consideiably. In this format the information is printed in light stippled grey covered by a heavy coloured overprint. Information may be uncovered using pens which contain a water

329

330 J. Marshall

soluble ink of similar colour to the overprint, the application of which reveals the underlying in- formation. In the non-examination situation, the use of an acetate strip of the same colour to reveal the information, rather than coloured pens, allows re-use of the programme.

Problem solving

The complete range of options offered to the participant attempts to overcome the problems of cueing inevitable in more closed structures which have been used elsewhere. The test commences with a brief statement about the presenting complaint (Fig. 1).

PATIENT MANAGEMENT PROBLEM

Mrs Dorothy May You have been called to the local hospital where friends have brought an unconscious 53 year old woman who has been visiting with them from interstate. She is wearing a Medic Alert bracelet stating that she is a diabetic on insulin therapy. NO FURTHER HISTORY IS AVAILABLE. Al l facilities are available and there are no resident medical officers. PROCEED TO SECTION A1 AND SELECT WHAT

ACTION YOU WOULD TAKE FIRST

FIG. 1. Brief statement on presenting complaint.

A branching programme is then provided, con- sisting of several major pathways or management sections from which directions are obtained to the sections where relevant data can be found. The initial pathway options are shown (Fig. 2), together with an example of the effect of selecting one choice (developing number 13).

Following direction from the pathway or manage- ment section to the clinical sections dealing with history taking, examination, investigation, special investigations, treatment and so on, the required data may be obtained in each of these sections as selected. An example is shown of such uncovered data related to history taking after the patient has recovered consciousness (Fig. 3).

On completion of each section the participant is invited to select appropriate action from the next pathway or management section (Fig. 4). The main pathway choices are allocated marks varying from f10 for the most appropriate choice to - 50 for completely inappropriate choices. The marks for each choice are allocated by a panel of experienced examiners. Each item of data is allocated a mark. In the final pathway section the directions ensure that the problem is eventually completed (Fig. 5). This figure shows the effect of revealing all possible pathway selections.

Items of information considered essential to solve the problem are allocated 1-3, less critical infor- mation f2, while minor information rates +l. Unnecessary information scores 0. On the other hand information revealed which is unnecessary and costly, or possibly even hazardous, scores negatively and may rate - 1 to - 50 dependent on its nature.

We consider that the efficient problem solver in performing the PMP will, after considering relevant initial hypotheses, eventually complete the exercise with the minimum information necessary for the particular problem.

It became obvious in the early years of the exam- ination that the most efficient problem solvers were

SECTION A1

Select from the list below what action you wish to take first. Unless otherwise instructed select ONE only. 10 11 12

13

14 15 16 17 18 19

Take detailed history 10 Perform detailed physical examination 11 Take brief history whilst making rapid 12 assessment of patient Perform rapid preliminary assessment 13 Go to Section C and on of patient completion go to Section A2 Perform surgery investigations 14 Order investigations 15 Give emergency treatment 16 Call in consultant 17 Send to hospital as a private patient 18 Refer to hospital as a public patient 19

FIG. 2. Initial pathway options.

Pro blem-solving 331

118 119 120 121 122 123 124 125 126

127 128 1 29 130 131 1 32

blood in sputum 118 blood in urine 119 blood in vomitus 120 bowel colic 121 nil bowel habits 122 normal breathlessness 123 nil bruising 124 buttock pain 125 chest pain 126 discomfort left upper

chest for 24 hours not varying with respiration

chemical, exposure to 127 claudication 128 colic-biliary 129 colic-bowel 130 colic-renal 131 colic-ureteric 132

FIG. 3. Erasures related to history taking after the patient has recovered consciousness.

SECTION A2

Select from the list below, what further action you wish to take. Unless otherwise instructed, select ONE only. Do not select any previously chosen. 20 21 22 23 24 25 26 27 28 29

FIG. 4. action.

Take detailed history 20 Perform detailed physical examination 21 Perform surgery investigations 22 Order investigations 23 Give emergency treatment 24 Commence elective treatment 25 Refer to consultant 26 Send to hospital as a private patient 27 Refer to hospital as a public patient 28 Review patient later in the day 29

Further pathways from which to select appropriate

not obtaining the top marks in what had been considered a problem solving exercise. The reason appeared to be that we had not really considered how the effective clinician works. We had reiterated the indoctrinated belief that ‘no history can be detailed enough’ and ‘no physical examination can be complete without a thorough systematic ap- proach’. In retrospect it was obvious that no family physician, nor indeed any specialist, really works in this way (Elstein et af., 1972).

In response to a few pertinent remarks made by the patient, early hypothesis formulation (one or more) occurs. As a result of this, a discriminating exam- ination takes place of the system which is relevant to the hypothesis. During this examination further questions may be prompted by some physical

finding. This is possible in the PMP in that once the history section has been entered, information can be obtained at any stage up to the final diagnosis. The outcome of such an approach will be either of two possibilities:

(1) the hypothesis will be reinforced and this will lead to confirmatory tests being performed; or (2) the hypothesis will be refuted and the problem will be reconsidered.

Developing the PMPs

As the PMPs were used it became obvious that refinement was necessary to overcome some of the anomalies which became apparent. Candidates’ scores were carefully monitored, both individually and as groups. The main areas of concern and the development aimed at relieving these difficulties are discussed below.

Limitations of PMPs

It was recognized that the PMP format can give a measure of the candidate’s performance in arriving at and confirming a diagnosis. This is the basis for inferring that he is capable of problem solving. The end-point is seen to be the referral of the patient or commencement of elective treatment on the assumption that the disgnosis has been reached.

In keeping with this restricted view of what the PMP could achieve as an assessment procedure, it was expected that candidates would score fairly consistently on different problems, but there would

332

SECTION A6

J . Marshall

~~

Select from the list below what further action you wish to take. Unless otherwise instructed, select ONE only. Do not select any previously chosen. 60

61

62

63

64

65

66

67

68

69

Take detailed history

Perform detailed physical examination Perform domiciliary investigations Order investigations

Commence elective treatment

Refer to consultant

Refer to local hospital

Refer to hospital as a public patient Review patient in 48 hours Review patient in 2 weeks

60

61

62

63

64

65

66

67

68

69

Go to Section B and on completion select again from Section A6 Go to Section J and on completion select again from Section A6 Go to Section H and on completion select again from Section A6 Go to Section 0 and on completion select again from Section A6 Go to Section F and on completion write your diagnoses in the boxes below. END OF PROBLEM Write your diagnoses in the boxes below. END OF PROBLEM Admitted. Select again from Section A6 Write your diagnoses in the boxes below. END OF PROBLEM Sister requests review. Select again from Section A6 Patient seeks treatment from another doctor. Write your diagnoses in the boxes below. END OF PROBLEM

FIG. 5 . Final pathways.

be reasonable variation based on familiarity with the disease conditions or because of experience in diagnosing presenting problems. The somewhat poor relationship between a candidate’s perfor- mance on different problems in the early years caused some consternation, and was identified as having two basic causes: (a) candidates reaching a correct confumed diagnosis with only essential information were not scoring as highly as some candidates who took more circuitous pathways and revealed a mass of data in the process; and (b) some candidates were unfamiliar with the PMP format and could not display their problem solving prowess, or may even have been insufficiently aware of some of the clinical problems presented.

The latter difficulties resulted in materials being used through the R.A.C.G.P. to familiarize can- didates with procedures, but the former required careful analysis of scoring procedures. We wished to fashion the scoring system so that the efficient problem solver - the one who arrived at the diagnosis by eliciting only the most important information - received the highest marks.

While recognizing that some teachers would disapprove of the use of penalty (negative) marks in the history taking section, it was decided to replace some of the minor scoring positive marks with zeros,

allowing ten zeros to accumulate a score of - 3, as these choices were regarded as time wasting and not helpful.

This system was adopted in 1973 (Table 1) and while little effect was produced, it was apparent by looking at the number of zero scores allocated that it was easy to determine which candidates had elicited superfluous data, yet they still scored better than the efficient problem solver who elicited only what was required to solve the problem.

TABLE 1. Analysis of marks scored-1973. Good problem solvers. 358 more efficient than 359

Surgical (220) Medical (320)

No.* t v e -ve Zero Total +ve -ve Zero Total

358 184 0 7 182 249 0 8 241 359 219 4 22 208 313 0 36 302

* In all ‘No.’ refers to candidates’ examination number

For example, candidate number 358 (Table 1) the more efficient problem solver, scored ‘relatively’ poorly. This candidate had uncovered all the information necessary to solve the problem with very little unnecessary information. Candidate 359 while eliciting the necessary information had also accumulated considerable information which,

Problem-solving 333

though not essential to solve the particular problem, had contributed excessively to his positive score. In the process, however, he had collected several zero and some negative marks, but these were more than compensated for by his extra positive score. Considerable thought was given to possible methods of correction of this anomalous state.

It was eventually decided that maximum marks should be applied to each individual section (history taking, examination, etc.), these marks being the total of the marks in each section considered necessary to solve the paiticular problem. The efficient problem solver could then gain all the marks possible in each of the various sections, while the more devious candidate who elicited superfluous data would not benefit in so doing, for even though he gained positive marks he could not exceed the maximum allowed and the more information elicited the greater would be the chance of accumulat- ing zero or negative marks which are deducted from the maximum section mark, rather than from the total of all the positive marks gained in that section. Moreover, a considerable number of previously zero scoring marks were now allocated - 1 , as it was thought that the majority of unnecessary in- formation elicited should be penalized.

This system was first applied in 1974, and appeared to overcome this anomaly. An analysis of the marks scored during 1974 (Table 2) and 1975 (Table 3) suggest that the most efficient ‘problem solvers’ who elicit only the data essential to solve the problem are now achieking the best results. Top marks (Tables 2 and 3) now indicate most efficient problem solvers.

TABLE 2. Analysis of marks scored-1974. Most efficient problem solvers now highest marks

Surgical (3 17) Medical (248)

No. 4-ve -ve Zero Total +ve -ve Zero Total

What the PMPs measure

While this development in the scoring procedures removed a potential source of difficulty in recogniz- ing the ability of an individual candidate, there was also appearing an interesting effect in the relation- ship of each group of candidates in their perfor- mance on both the surgical and medical problems. This effect could be seen to be operating in two ways: (a) mean scores on the two problems (PMP 1- surgical, and PMP 2-medical) were fluctuating, and some control on the degree of difficulty of the cases was felt desirable ; (b) the correlation between the two PMPs increased from 1973 on, perhaps reflecting the increasing rzliability in the candidates’ results being achieved as improvements were intro- duced.

Cases selected for use varied in their familiarity to the candidates, being acute appendicitis and obstructive jaundice in 1974, for example. The possibility that variation in performance could be influenced by the factual material involved, or even by the order in which the tests were performed, was investigated.

In 1975, it was decided to design the two problem solving tests based on commonly presenting con- ditions such as chest pain, jaundice or breathlessness so that they would be of more comparable levels of difficulty. It was also arranged that half of the candidates would attempt one of the problems fust, with the remainder the other problem 6rst. Analysis subsequently indicated that the order of performance of the tests was not important. Results for all candidates for all years 1972-1976 are summarized in Table 4.

TABLE 4. Analysis o f marks in relation to the order of performance of two tests by candidates

Mean SD SE n t-test Probability

245 263 0 1 263 242 0 2 241 247 275 12 1 263 236 2 2 233 307 285 10 1 275 231 0 1 231 511 272 6 2 265 241 0 0 241 410 262 20 0 242 244 3 8 239

TABLE 3. Analysis of marks scored-1975. Most efficient problem-solvers

Surgical (260) Medical (260)

No. +ve -ve Zero Total +ve -ve Zero Total

286 248 0 5 246 250 0 3 249 306 250 0 2 249 249 0 6 247 322 250 1 2 248 245 3 3 241

1972 PMP 1 PMP 2

PMP 1 PMP 2

PMP 1 PMP 2

PMP 1 PMP 2

1973

1974

1975

63.51 63.58

77.01 70.08

63 .84 78.28

82.89 83.73

12.26 1 . 3 1 88 -0 .03 0.973 18.86 2.01

15.02 1 .32 130 3.91 <0.001 18.54 1.63

14.63 1.38 113 -10.35 <0.001 13.56 1.28

10.48 0 . 8 0 171 -1.03 0.306 10.16 0 . 7 8

1976 PMP 1 70.50 14.43 1.05 188 -6.58 <0.001 P M P 2 76 .12 11.31 0 .83

334 J. Marshall

Only in 1972 and 1975 were there no significant differences in the performance of the groups of candidates on the two PMP problems as reflected in their means. However, the correlation coefficients for each year between the individual candidate’s performance on the two problems shows an im- proved relationship from 1973 (1972, r = 0.35; 1973, r = 0.29; 1974, r = 0.45, 1975, r = 0.47; 1976, r = 0.61).

This might indicate that there is increasing agreement between the two problems on some factor common to problem solving in medicine, but does not show sufficient agreement to allow only one problem to be used on the basis that it will measure sufficiently clearly a candidate’s problem solving ability. The essential background knowledge of specific cases must still be considered an important factor in the use of this type of assessment procedure.

Summary

It is recognized that no problem solving can take place without a background knowledge. As the highest marks in PMPs are now being consistently achieved by those candidates who arrive at the correct diagnosis without accumulating excessive information and at the same time avoiding irrelevant or incorrect data the PMP may be measuring efficiency in problem solving ability. This would appear to be substantiated by the fact that reasonable correlations are obtained between each candidate’s remarks in problems of widely differing system disease patterns.

The mark allocation is such that the problem solving ability relates to the field of general practice and no detailed specialist knowledge of any parti- cular disease pattern is being measured. In measuring problem solving ability it is advisable to allocate a maximum positive mark for each section, this mark being the total of that awarded for information considered necessary to solve the particular problem. Any negative marks scored must be taken from the maximum positive mark allowed and not from the total positive score obtained, which may be con- siderably more than the former. In this way the efficient performer is appropriately rewarded whilst the more devious data gatherer i s appropriately penalized.

Acknowledgments

The author wishes to acknowledge the many people who contributed helpful advice and criticism, in particular Dr Wes Fabb and Mr Neil Paget, the fatter providing the statistical analysis.

References ELSTEIN, AS., KAGAN, N., SHULMAN, L.S., JASON, H. &

LOUPE, M.H. (1972) Methods and theory in the study of medical inquiry. Journal of Medical Education, 47, 85.

FABB, W.E. (1967) Personal communication. MCGUIRE, C.H. & BABBOTT, D. (1967) Simulation technique

in the measurement of problem solving skills. Journal of Educational Measurement, 4, I .

NEWBLE, D. (1975) Personal communication.