Nurse Educahon Tcdq (1989) 9,s12 0 Longman Group UK Ltd 1989

02606917C3Y/0009-0003/$10.00

Student assessment in basic nursing education in Scotland

Annie T Altschul and Helen C Sinclair

This paper presents some of the findings of a 4-year-study of the examination system for student nurses entering basic nursing courses offered by all Colleges of Nursing and Midwifery in Scotland. These modular courses, first implemented in 1982, involve students in a series of internal (college) assessments and two external (national) examinations.

Results for 2742 students on 16 764 written assessments were analysed. The hypotheses that there would be high correlation between different modular results in internal assessments, and between results in internal assessments and external examinations were not supported.

Implications for possible changes in the examination system are discussed.

INTRODUCTION

The assessment procedures in basic nursing education in Scotland involve learners in two main sets of written examinations: ‘internal’ or college examinations, set and marked by teach- ing staff in the Colleges of Nursing and Mid- wifery and ‘external’ or national examinations set and marked by examiners appointed from among the same college staff by the National Board for Nursing, Midwifery and Health Visit- ing for Scotland (NBS).

In contrast to recent English national basic nursing examinations Scotland has continued to rely on essay type questions as the main form of written assessment in its national examinations.

Annie T Altschul CBE FRCN RGN RMN RNT BA MSc Professor Emeritus, Helen C Sinclair BSc RSCN RGN RM RNT Lecturer, Department of Nursing Studies, University of Edinburgh, Adam Ferguson Building, 40 George Square, Edinburgh EH8 9LL (Requests for reprints to ATS) Manuscript accepted 27 April 1988

N.E.T ~~ B

It is well known that there are difficulties with inter-marker reliability in essay based assess- ment (Hartog & Rhodes 1936). The former General Nursing Council for Scotland in an attempt to increase reliability developed and used a computer programme to ‘standardise’ examination marks awarded to candidates by different examiners. In the national examin- ations as they were then, any one candidate’s answers to questions on a particular paper were assessed by a single examiner. Because exam- iners differed in the criteria they used when awarding marks, for ‘stiff markers an upward adjustment of marks had to be made, for ‘lenient’ markers a downward adjustment.

By 1982 curriculum development in basic nursing education in Scotland had resulted in the introduction of new modular courses and consequently, different assessment procedures. Whereas the former assessment procedures allowed college staff to determine the points in the curriculum at which internal examinations would take place. the new procedures required

3

4 NURSE EDUCATION TODAY

learners to be assessed in clinical practice and examined on the theoretical content in each module in both first and second level courses. As a result the required number of times each learner had to sit internal written examinations increased from none or one (the intermediate examinations for students) to at least four in second level courses and eight or more in first level courses. A pass grade in modular assess- ment, including written assessment, was re- quired for permission to proceed to the next module.

The national examinations were also affected by curriculum revision. Though course specific national examinations continued to take place towards the end of the various courses, an additional national examination was inserted half way through the 3 year first level courses at the end of the first stage. This ‘Stage One’ examination is common to all students, irrespec- tive of the part of the register for which the course was designed. A pass in this examination was required for progression to the second stage (into Module 5 in most colleges).

The format of the national examination papers was designed to reflect the modular nature of the courses. The papers were divided

into sections containing two alternative ques- tions, most of them reflecting the content of particular module types e.g. care of the elderly.

This change in format also prompted re- organisation of marking procedures. Instead of one examiner marking all answers to questions in a paper written by a candidate, four exam- iners are now involved in awarding marks to each candidate, one for each section tackled. Since it was assumed that this alteration would in itself help to increase reliabilty the ‘standard- isation’ computer programme was no longer considered necessary in the new national exam- inations.

THE AIMS OF THE RESEARCH

The study, supported by the NBS, was designed to provide future planners in basic nursing education with an evaluation of the new examin- ation procedures.

Recently there has been an emphasis on two distinct types of ‘goal free’ evaluation: sum- mative and formative (Striven 1967). Summa- tive evaluation involves the collection of data at the completion of a course and the production of an objective report. Formative evaluation, in contrast, provides interim information for those running the courses to assist them with adjust- ments and/or improvements. The research aimed to provide both.

Seven hypotheses were constructed and col- lection of detailed information about various aspects of the written assessment procedures was planned. This paper deals only with the quanti- tative analysis of the data about first level written assessment and the hypotheses discussed are the following:

1. that the learners’ success in internal examin- ations will correlate highly with success in national examinations,

2. that internal examinations will have a very high predictive value for the first stage national examination, and that consequently the national first stage examination will become superfluous,

3. that performance in the national final exam- ination will not be highly correlated with performance after individual modules if the final examination tests different attainments.

RESEARCH METHOD

Letters were initially sent out to all colleges of nursing and midwifery in Scotland requesting co-operation with data collection. After consul- tation and exploration of how examination results were normally recorded, data collection forms were designed and distributed to the colleges. Information was collected about the timing, form and content of modular assess- ment, and about marks for the components which counted towards overall modular results. Access to the marks awarded to learners in national examinations was granted by the NBS. What exactly constituted pass/fail in terms of marks awarded was ascertained. Specimens of college and NBS question papers attached to

NURSE EDUCA’I‘IO~ TODAY 5

each set of examination results were obtained. 160) were excluded from the study. Originally, it was believed that retrospective

data were available and that the first four intakes into the new courses in 1982 could form the study group. After piloting some of the 1982 data it was decided to study the written assess- ment for the entire cohort of pupils and pre-

registration students who entered basic nursing education courses offered by the colleges in 1983.

Students were divided into four groups according to their date of entry. For Group 1, (those entering first level courses between January and March 1983) information became available for 769 students. The size of the groups diminished during the year to 635 for Group 4 (students entering between October and Decem- ber 1983).

When the forms were returned and infor- mation collected from the NBS, data were coded and entered into the computer. From then on only individual index numbers were used to identify learners. A flow chart allowed the researchers to keep track of the returns for the four years of data collection.

While the research proceeded, important policy decisions about nursing examinations were taken by both the English and Welsh National Boards, who decided to discontinue their national examinations. Project 2000 dis- cussion papers were circulated with proposals for UK wide curriculum revision and the aban- donment of second level training (United King- dom Central Council for Nursing, Midwifery and Health Visiting 1987).

It was estimated that students should have completed NBS Stage One examinations by April 1985 and NBS final examinations by 1986, but inevitably some students were late reaching these examinations. The results of seven Stage One examination diets were searched for mem- bers of the cohort, but only five diets of final examination results. Of the cohort members 75% successfully completed NBS fmal examin- ation by October 1986. Taking into account delays and re-entries it is likely that more cohort students completed in 1987 - after the data collection ceased.

Several interim reports of this research were made available to the Colleges of Nursing and Midwifery in Scotland and the NBS. These were intended to provide feedback on early results (Altschul & Sinclair 1986; 1987a).

In total, the number of overall internal modu- lar results (n = 11,789) and NBS examination results (n = 4937) which became available for analysis was 16,764. Table 1 shows the number of student results in the component parts of the overall modular results. It can be seen that for most students formal timed examination, mainly consisting of structured essay type questions, was the only method of assessment. they experi- enced. Projects e.g. nursing care studies and other forms of assessment e.g. multiple choice questionnaires were much less common and mainly used in conjunction with formal examin- ations in modular assessment.

FINDINGS FOR FIRST LEVEL EXAMINATIONS

All colleges agreed to participate and provided data on written assessment, but not for all modules or for all students. Although some amalgamation of colleges occurred during the data collection period, returns from the original 21 colleges continued to be identifiable. Data about written assessment for 2724 elegible pre- registration students became available. This rep- resented just over 90% of all pre-registration students indexed in 1983 (n = 3037) (Hutchings 1985), since students who were graduates and those on nursing degree courses (approximately

Examination questions and results

Formal college examination papers contained between 2 and 8 questions. Candidates were

Table 1 Number of First Level Results as Components of Modular Assessment

Component Number of Results

Formal Examination 10 364 Project 2941 Other Written Assessment 913

6 NURSE EDUCATION TODAY

105

s

5 90

1

E f 75:

ti 60 -

45

- Module1

- Module2

- Module3

- Module4

0

0 5 101520253035404550556065707580859095100

Fig 1 Distribution of Marks by Module Order in Stage One

required to answer anything from 2 or 3 ques- tions without choice to 3 or 4 questions out of 8. The pattern within Colleges was fairly consis- tent. Marking scales, however, differed. For the purpose of analysis, and in keeping with prac- tices in most colleges and the NBS, percentage scores were used with a pass equalling 50%. Where necessary, college scales with different pass marks were converted to allow comparison. For one college such conversion proved to be impossible and results from that college were excluded from the analysis.

Figures 1, 2 and 3 show the distribution of modular assessment and NBS examination results for students in Group 1.

The distribution was similar for all student groups. For all internal module results the means ranged from 66% to 7 1 %, and there was a negative skew (the median being higher than the mean). NBS examination results were normally distributed, with means and median around 63% for Stage One examinations and 61% for final examinations.

Very high scores occurred more frequently in college assessments than in NBS examinations. Failure rate was low in all examinations but slightly higher in NBS examinations than in college module assessments.

Because the researchers were interested in whether a predictive value could be attributed to modular examinations (see hypotheses) it was decided to follow some students through all their examination results. To make this easier, per- centage scores were grouped into six categories as follows:

Low fail Less than 45% Marginal fail 45% to 49%

Marginal pass 50% to 55% Safe pass 56% to 69%

Good pass 70% to 75% Excellent pass More than 75%

Figure 4 shows the different percentages of student’s results falling into each category in college modular assessments, and in NBS Stage

NURSE EDUCATION TODAY 7

loo-

80 -

60 -

1

Module 5

Module 8

Module 8

0 5 101520253035404550556065707580859095100

Fig 2 Distribution of Marks by Module in Stage Two

STAGE1 -

FINAL EXAMINAI’ION -

Fig 3 Distribution of Marks for NBS Examinations

One and final examinations. It can be seen that a higher proportion of students obtained ‘excel- lent’ passes in college assessments compared to NBS examinations. This was a consistent finding across groups, but not across colleges.

For students near the borderline between grades, and particularly between pass and fail, low reliability may be very important. In fact very few students fell consistently into the margi- nal groups. Students who failed any one of the college modular assessments or NBS examin- ations were followed up as far as this was possible (n = 480). Some of these students may well have discontinued. However, of the 298 traced among the NBS final results, 37 (7.7%) failed more than one of the examinations; some scored excellent marks in other examinations (Figs 5 & 6).

Far too many candidates scored excellent passes in college examinations to allow for indi- vidual follow-up. The results for the much smaller number with excellent passes in NBS examinations were traced through all modular examinations. These students’ modular perfor-

8 NURSE EDUCATION TODAY

100

:obge 1st stage NBS 1st stage College 2nd stage NBS Final

W Excellenl Pass

GoodPasa

0 Safe Pas8

q Marginal Pass

q Marginal Fail

n Low Fail

Fig 4 Students’ Exam Results On First Entry In Categories of Marks

mance rarely foreshadowed their excellent per- formance in NBS examination.

It seemed of interest to explore whether higher correlations could be found when investi- gating how candidates performed in specific parts of the course.

Colleges differed in the order in which differ- ent module types were arranged. Results ana- lysed by module order and module type showed very slight differences in failure rates.

Table 2 shows that there was a gradual decrease in failures from Module 1 to 5, perhaps because of attrition of some of those who failed. Module 6 (Stage 2) was for students predomi- nately a medical nursing module and Table 3 shows that the failure rate in Stages 1 8~ 2 was highest in medical nursing.

Qualitative analysis of the type of questions asked, reported elsewhere (Altschul & Sinclair 1987b), shows that Module 6 is the one in which a

Table 2 Number and Percentage of Passes and Failures at First Entry to First Level College Examinations by Module Order

Module Pass order n %

Fail n %

Total n %

First Second Third Fourth Fifth (Stage I) Fifth (Stage 2) Sixth Seventh Eighth

Total Not Known Missing

1996 95.7 1952 96.3 1817 96.7 1345 98.8

26 -

930 99.2 1522 94.6 900 96.3 300 92.6

IO 788

89 4.3 2085 100 74 3.7 2026 100 61 3.3 1878 100 25 1.8 1370 100

4 - 30 100 7 0.8 937 100

87 5.4 1609 100 35 3.7 935 100 24 7.4 324 100

406 11 194 61

239

Total 11494

N.B. (II Module 8: Numbers of pass/fail results known are small although other information is available (2) Not Known: Result in percentage marks enterd, but not whether candidate passed or failed

Missing: Neither mark or pass/fail was known, but other information was given

number = 480

298 TRACED AMONG

NBS FINALS RESULTS 62% 182

UNTRACED AMONG NBS FINAL RESULTS

BY OCTOBER 1986

Fig 5 Students Who Failed One or More Examinations

NURSE EDUCATION TODAY 9

demand for a higher order or conceptual

thought is made for the first time. This may

partly account for the upturn in failure rate.

For care of the mentally handicapped and sick children modules the numbers of students in-

volved in the data were too small to calculate the percentage of failures.

A detailed analysis of correlations and an analysis of variance was carried out on the results

of a subgroup of 499 students for whom almost

complete information was available. The highest correlations were those between the care of the

elderly section in the NBS Stage One examin-

ation and the care of the elderly module results (Rho = 0.21, p < 0.001) and between the section

on the care of the mentally ill and the corres- ponding module results (Rho = 0.2, p < 0.001).

Differences between colleges were statistically significant for college modular examinations, but not for NBS examinations.

During the first stage of the courses the part of

the register for which some of the students intended to study was ‘not yet known’. During

Table 3 Number and Percentage of Passes and Failures at First Entry to First Level College Examinations by Module Type in Stage One and Stage Two

Stage One Stage Two Total

Module type Pass Fail Not known Pass Fail Not known Pass Fail Not known

n 1695 44 10 0 0 0 1695 44 10 Elderly

Medical

Surgical

Medical/Surgical

Mental Illness

Mental Handicap

Children

Mixed (I)

Management/

% n % n % n % n % n % n % n % n

97.0 1833 94.7 1745 93.8

32 -

1442 98.2 191

-

164

2.5 0.5 91 11

4.7 0.6 85 30

4.6 1.6 5 0 - -

19 8 1.3 0.5

3 4 - -

6 0 - -

0 0 - -

0 0

0 0 0 97.0 2.5 0.5 1129 76 3 2962 167 14 93.5 6.3 0.2 94.2 5.4 0.4 808 34 2 2553 119 32 95.7 4.0 0.3 94.4 4.4 1.2

35 3 1 67 8 1 -

336 98.2

42 -

175

- -

5 1 1.5 0.3

1 1 - -

7 0

0

0

812 99.0

- -

3 5 0.4 0.6 24 0

- - -

1778 24 9 98.2 1.3 0.5 233 4 5

- - -

339 13 0 - - -

812 3 5 99.0 0.4 0.6 309 24 0

High dependency % - - - 92.8 7.2 0 92.8 7.2 0

Total n 7102 253 63 3646 153 13 10 748 406 76 % 95.7 3.4 0.9 95.6 4.0 0.4 95.7 3.6 0.7

(1) Mixed module varied, but commonly children/maternity/community. N.6. Not known: result in percentage mark was entered, but not whether candidate passed or failed. Missing

values excluded.

10 NURSE EDUCATION TODAY

number = 480

Fig 6 Overlap of Modular and NBS Examination Failures

the second stage, this information was available for all students. Table 4 shows the percentages of failures and excellent passes for the NBS final examinations for different parts of the register. Results for part three (RMN) are much better than for the other parts of the register. It can be seen that in spite of the large numbers of data, the number referring to part five (RNMH) and part eight (RSCN) are too small to draw con- clusions, even for NBS examinations. For most analyses of modular assessments by colleges the numbers were much too small.

DISCUSSION

The schemes of basic nursing education on

which colleges of nursing and midwifery em- barked collectively in 1982 is an ambitious one, designed to provide all students with a broad base of theoretical tuition and clinical experience in the first stage and the opportunity to study in more depth for different parts of the register in the second stage.

For the first time, as a result of this research, colleges have access to detail other than pass or fail about their own students’ performance in national examinations. Although the research results are inconclusive, the main findings should provide a data base for discussion of the many issues which this research has raised. Details of the findings were supplied to colleges of nursing and midwifery, each of which can compare its own procedures and students’ results with those of other colleges collectively.

The modular curriculum had as one of its objectives the integration of theory and practice. The requirement that students successfully com- plete each module before moving on to the next, generated a very large amount of assessment. It was left open to colleges to decide on the format the assessments should take. As it was also one of the objectives that there should be an even flow of students through the system, the timing of assessments within each module had to be con- sidered.

Assessment early in the module, which occur- red in some colleges, seems incompatible with the intention of integrating theory and practice and sets narrow limits to what can be assessed. It does, however, allow for resits within the period of the module and upon successful outcome allows the resit candidates to progress in the course with their own peer group. Failure in

Table 4 Excellent Passes and Failures in NBS Final Examinations for Each Part of the Register

Register Part

One (RGN)

Three (RMN)

Five (RNMH)

Eight (RSCN)

n % n % n % n %

Fail Excellent Pass Others Total

81 46 1457 1584 5.1 2.9 92 100 4 67 374 445 0.9 15.1 86 100 5 6 90 101 5 6 89 100 3 11 114 128 2.3 8.6 89.1 100

NURSE EDUCATlOh’ TODAY 11

assessment at the end of the module, which was when assessment took place in most colleges, resulted in the requirement for the student to repeat the module and to join a different group of students.

Colleges seemed to differ in their policy of asking students to discontinue the course after failure of modular examinations. It is not known how many students discontinued, either at the request of the college or of their own free will. It is, however, interesting to note that for some students information, including resits, became available for as many as eleven modules (three more than normal in the colleges concerned).

Examinations are time consuming for exam- iners and it is generally believed that examin- ations are stressful to candidates. It should be noted how frequently students are having to face a situation in which their teachers are making decisions affecting their future careers. For the weaker students this may contribute substan- tially to stress and anxiety.

While it is acknowledged that some teachers believe in the value of frequent examinations as part of the formative assessment of students’ progress and as a guide to the effectiveness of their teaching, the linear form of the modular courses mean that students are also examined frequently for selection (or rejection) purposes.

This research aimed to investigate whether any reduction in the frequency of examinations might be contemplated. High correlations be- tween internal assessment and NBS examin- ations might have suggested that some assessments could have been dispensed with. In the event, correlations turned out to be low. Failure rates, however, were also low, and the pass rates on re-entry to all assessments were high. Thus it was impossible to identify any particular module or NBS examination which was markedly different from others in terms of the student outcomes.

Whether it is really imperative that each student should pass every single one of the assessments and whether success by the end of a module should really be a necessary condition for proceeding to the next module warrants discussion. The results show that students per- form unevenly in different modules and that the

outcome of examinations in earlier modules has low predictive value for later ones.

It might be that interest and motivation for learning in different module types affected the outcome, or that personal affairs intervened. Fransson (1977) has shown how factors such as interest, relevance and anxiety can influence the learning process and outcome. It is also possible that earlier knowledge might be better under- stood in the light of subsequent experience and teaching. The requirement to repeat a module may be counter-productive. Hypothesis 1, that modular results would be highly correlated, was not supported.

The format of the NBS First Stage examin- ation was such that results in five of the six sections could be compared with the college assessment results for specific module types. The research findings did not support Hypothe- sis 2, that internal examination would have a very high predictive value for the national first stage examination. Holding college and NBS First Stage examinations constant the corre- lations between students’ results matched for module and section type, and between overall NBS results and modular results, were low.

The remaining section of the NBS First Stage examination contains questions on anatomy and physiology, subjects which form part of every module in the first stage. Therefore, the results in this section were not correlated with module results. In fact the failure rate for individual questions on anatomy and physiology in NBS First Stage examinations (Section I) reached 28.7% in spite of the low overall failure rate. It was not possible to compare this finding with the college results on questions on anatomy and physiology because of the variety of assessment methods used, including multiple choice ques- tionnaires.

Abolishing the NBS First Stage on the grounds of high correlations with internal exam- inations could not be recommended.

Low correlations were also found between module results and NBS final examinations as stated in Hypothesis 3. If the outcome of college assessment were to reveal integration of theory and practice within modules, and the NBS examination were to assess a different kind of

12 NURSE EDUCATION TODAY

integration across modules, low correlations might be expected between modular and NBS results. This might support an alternative hypo- thesis that different attainments were tested in modular assessment and NBS examinations (as referred to in Hypothesis 3 in relation to national final examinations).

When it came to comparing module perfor- mance in the second stage with performance in the NBS final examinations it became difficult to hold examinations constant and obtain large enough numbers especially for students study- ing for parts three, five and eight of the register. For this reason individual students who failed in one or more of either module assessment or NBS examinations were traced and other available results extracted. A similar exercise was under- taken for those who had ‘excellent’ passes in NBS final results. Very few of the students who reached the NBS final examinations failed or consistently obtained excellent passes in both NBS final examinations and modular assess- ments.

In any examination the overall outcome for any one candidate can be substantially influ- enced by very high or very low marks in any part of the examination (Rowntree 1977). For this reason, in a modular type of education, it seemed important to investigate the influence which any specific module or any specific section of the NBS examination might have on examin- ation results.

The decision to study the examination results of a large number of students and to include all colleges and all modules was taken in order to be able to investigate the influence of those results which derive from examinations where the numbers are smallest. No such influences were found. For example, it was found that in the NBS final examinations questions on ‘Aetiology, Pathology and Treatment’ and on ‘Nursing (Wider Aspects)’ i.e. Sections One and Two of the NBS final examination in general nursing, had a high failure rate relative to other sections (between 9% and 14%), but that the overall pass rate was not influenced.

Results of modules and NBS examinations relating to the nursing care of sick children and

draw conclusions, in spite of data from a whole years cohort.

The findings also show that colleges of nursing differ in the way they conduct written assessment, in the distribution of the marks they award, but that each college is consistent over most of the modules and student intakes. It would seem that some form of external examin- ation of students should be retained to provide standards and prevent the consolidation of different assessment and marking practices in colleges. In addition, in order to assess the cumulative outcome of the modular training as opposed to the content of specific modules at the least a final or end of course assessment is required.

Acknowledgement

This research was supported by a grant from the Chief Scientist Office of the Scottish Home and Health Depart- ment.

References

Altschul A T, Sinclair H C 1986 Report on the examination of candidates who entered second level training in 1983; second interim report. Unpublished report. University of Edinburgh

Altschul A T, Sinclair H C 1987a Report on the examination of candidates who entered first level training between January and March 1983: third

interim report. Unpublished report, University of Edinburgh

Altschul A T, Sinclair H C 1987b Nursing examinations in Scotland: an obstacle course towards registration. Paper presented at the International Nursing Research Congress, University of Edinburgh, July 1987

Fransson A 1977 On qualitative differences in learning IV - effects of motivation and test anxiety on process and outcome. British Journal of Educational Psychology 47: 244-257

Hartog P, Rhodes E C 1936 The marks of examiners. London: Macmillan

Hutchings M J 1985 Report on statistics. Unpublished paper. National Board for Nursing, Midwifery and Health Visiting for Scotland, Apric 1985 ’

Rowntree D 1977 Assessing students: how shall we know them? London: H&per & Row

Striven M 1967 The methodology of evaluation in perspectives of curriculum evaluation, editors, R Tyler, R Gange, and M Striven New York: Rand McNally

United Kingdom Central Council for Nursing, Midwifery and Health Visiting 1987 Project 2000: The Final Proposals. Project Paper 9, London:

of the mentally handicapped were too few to UKCC.