Eraut Report into Surgical Training / ISCP - Full

7/29/2019 Eraut Report into Surgical Training / ISCP - Full

1/69

i

Preface to the ISCP Evaluation Report

The case for a major review of surgical training in this country had been steadily building

since the late 1990s. The introduction of the Calman reforms during 1996 saw

improvements to higher surgical training but left basic surgical training unreformed.

Many aspiring surgeons spent years waiting to enter specialty training, going from job to

job, often referred to as the lost tribe.

Additionally other external factors were starting to impact on traditional surgical trainingas a whole and were increasing the pressure for change, for example:

The European Working Time Directive (EWTD); Increasing public expectations for accountability and transparency; and New working practices and changes to service delivery.

The decision to review existing surgical curricula began in 2002. In 2003 Modernising

Medical Careers provided additional impetus to develop new curricula. The Chairman of

JCHST, now JCST, and the Specialist Advisory Committee (SAC) Chairs, together with

their delegated editors, led the process of curriculum creation and the Intercollegiate

Surgical Curriculum Project (ISCP) was established in March 2003. Practising surgeons,

trainees, educationalists, and other specialists were involved in all aspects of curriculumdevelopment.

The curriculum was designed to integrate four key domains in surgical practice: clinical

judgement, technical and operative skills, specialty-based knowledge, and genericprofessional skills. For the first time, the evolving curriculum articulated stages of

training and the standards to be achieved, in the four domains, at each stage. This was an

innovative approach to surgical education and training and one that was very differentfrom existing practice. As the curriculum developed it was decided to integrate the

curriculum with a web based training management system that would promote andsupport good educational practice.The curriculum development went through a pre-pilot and pilot phase prior to launching.

At an early stage the services of Professor Michael Eraut, an expert in professional and

workplace learning, were enlisted to act as a participant observer who would produce an

independent report at the end of the pre- pilot phase(http://www.iscp.ac.uk/Documents/EvaluationReportPhase2.pdf). The findings of thereport highlighted the importance of undertaking contemporaneous research and

evaluation.

The ISCP Evaluation Task Group was established under the chairmanship of Professor

Irving Taylor in 2006. It reported to the Curriculum Project Management Group chairedby Professor Gordon Williams. A grant of 50,000 was allocated from the ISCP

development budget to fund small research projects to contribute to the overall debate.

Six areas of research were identified:


2/69

ii

access to appropriate experience for trainees; provision of appropriate resources and support for trainers; the effectiveness and cohesion of the underpinning infrastructure, for both provision

and regulation;

organisational issues around time and resources that impact on the feasibility ofdelivery of the new curriculum;

the effectiveness of educational and assessment tools and resources developed as partof the project;

the effectiveness of the change strategy for implementing the new curriculum.It was intended that the research projects would be carried out by the deaneries in

partnership with their schools of surgery and an academic institution that would act as a

research agent. Deaneries were asked to bid to conduct a research project against one

of the six research areas. Four projects were chosen, as shown below.

Organisation Title of ProjectKent Surrey and Sussex Deanery An evaluation of the KSS methods for introducing ISCP through

the establishment of Surgical faculty groups within TrustsSchool of Postgraduate Medical and

Dental Education, Cardiff University

and Wales Training School of

Surgery

The Intercollegiate Surgical Curriculum: an evaluation of the

online learning agreement and web-based resources

University of Southampton Do Current Surgical Training Posts Provide AppropriateExperience and Support for Future Surgical Trainees?

Warwick Medical School The Effects of Changes to the Modern NHS upon Continuity ofPatient Care, Surgical training and Overall Patient Care

The original time frame for data collection was February July 2007, however this was

adjusted in the light of the project partners proposals. The research therefore spanned

both the pilot phase and initial implementation. Professor Michael Eraut was chosen asthe external academic advisor across the various research groups to provide advice on the

main themes of the project overall, and to facilitate cross-fertilisation of ideas.

Alongside the ISCP funded research, both the ISCP and PMETB conducted evaluation oftrainees experiences. Professor Michael Eraut was commissioned to produce an overall

summary on behalf of ISCP drawing on the results of the ISCP research projects and data

from the ISCP and PMETB surveys.

Professor Irving Taylor Francine Alexander

Chairman, ISCP Evaluation Task Group Interim Head of Education

RCS England


3/69

iii

Acknowledgements

This report is based on evidence collected by a large number of people. The quantitative

data comes from four sources: The JCST Quality Assurance team led by Nigel Stripe, thePMETB website, the ISCP website managed by Jeremy Brooks-Martin and an

opportunistic study of London trainees attending Anatomy days at RCS by John Masih.

The qualitative evidence comes from four research studies commissioned by the ISCP

Evaluation Task Group, chaired by Professor Irving Taylor, following an invitation to all

the Deaneries. Given the wide variations in trainees experiences, it has been important to

present the experiences of trainers and trainees in different parts of the ISCP partnership

both collectively and individually through their own voices. For this data, I am indebtedto the fine research undertaken by these studies, especially to their leading researchers:

Robert Padwick, Deborah Markham and Neil Johnson at WarwickMary Seabrook, Pam Shaw and Mary Hayes at KSS

Stephen Brigley, Louis Fligelstone, Lynne Allery, Janet MacDonald and Lesley Pugsleyat the Wales postgraduate deanery

James Gilbert, Karen Nugent and Debra Humphris at Southampton

I have referred to these studies when they were the primary source for one or more pages,but not on every occasion when their work was used because that would have made the

text difficult to follow; and there were also places where the context required a rather

different wording than the original. I take responsibility for such changes and have

checked with the authors that they are not unhappy with my interpretations. I also gained

many new insights through elite interviews with experienced surgical educators, who alsovetted my use of their interviews. All the data collected by all these sources promised

anonymity to their informants.

An important part of my contribution has been an attempt to provide a bigger picture than

could be provided from any single piece of research in order to show how variations can

be incorporated into a more complex but understandable picture. Thus my analysis seeksto show how different findings connect with each other, and how we might plan policy

pathways that address some of the considerable difficulties that confront surgical

education today. In seeking this more ambitious interpretation of my brief, I have been

greatly helped by three former members of the Raven Department of Education at RCS

Eng: Natalie Briggs, Andrea Kelly and John Masih. Finally, I would like to thank thosemembers of education related ISCP and RCS committees, especially Professor Irving

Taylor and Francine Alexander who gave me considerable advice and support during the

final period of my evaluation.

None of the helpful people listed above are responsible for my final text and

recommendations; so any mistakes, misunderstandings or disagreements are my own

responsibility and critical comments should be sent to myself alone.

Michael Eraut, Professor Emeritus, University of Sussex


4/69

iv

CONTENTS

Preface Page i

Acknowledgements Page iii

Contents Page iv

Summary Page 1

Introduction Page 7

Data sources for Pilot Evaluation Page 7

Changes in the Surgical Education Context outside ISCP Page 9The Impact of MMC and EWTD Page 9

Less time for training for both trainees and trainers Page 10

Providing continuity of patient care Page 11

Providing continuity of training Page 13

Organisational factors and targets Page 14

Early implementation of the ISCP curriculum Page 16Educational Supervisors and Learning Agreements Page 16

The ISCP website Page 19

Initial response to ISCP Page 21

Learning contexts and access to learning Page 27

Elective Surgery Page 28

Emergency Surgery and Trauma Page 30

Clinics for Outpatients Page 31Ward work Page 32

Simulators, models, endoscopy and recordings Page 33Formal educational activities Page 34

When is Service Work a learning opportunity? Page 35

Apprenticeship, informal learning and coaching Page 37

Competence and Assessment Page 41

Portfolios Page 44

ARCP (formerly RITA) Page 44

Teamwork and relationships Page 46Handover of patients Page 49

Support for Trainers Page 51

Distributed Apprenticeship and Organisational Practices Page 55

Risk analysis andnew organisational practices Page 55

The training of surgical trainers and surgical teams Page 57

References Page 58

Recommendations Page 59_____

Appendices A to E Pages 60 -64


5/69

v

TABLES

Table 1 % of validated trainees with an AES and a learning Agreement Page 16

Table 2: Meetings about progress and formal assessment in General

Surgery Page 17

Table 3: RCS Quality Assurance survey data on Supervision Page 22Table 4: PMETB data on clinical supervision, informal feedback and

ethics Page 22

Table 5: QA data on Appraisal and Feedback Page 23

Table 6: The balance between constructive and undermining Feedback Page 23Table 7: From whom do trainees get their support, and how much? Page 24

Table 8: QA data (1-5 scale) on Induction Page 24

Table 9: QA data (1-5 scale) on Response to Trainees Concerns Page 25

Table 10: QA data (1-5 scale) on Organisational Issues Page 25

Table 11: Distribution of trainee time across patient contexts Page 27Table 12: Issues relating to Learning in the Workplace Page 28

Table 13: Availability and Quality of Formal Educational Opportunities Page 34Table 14: Formal Teaching and Audit Page 34

Table 15: The Informal Audit of critical events and near misses Page 34

Table 16: Frequency of on the spot discussions of clinical/professional

episodes Page 38

Table 17: Frequency of discussions about a trainees work Page 38

Table 18: Frequency of discussions about a trainees general progress Page 39Table 19: QA data (1-5 scale) on Assessment Page 43

Table 20: Trainee use of Portfolios and Logbooks Page 44Table 21: Handover Arrangements Page 50

Table 22: Frequency of topics discussed at KSS Faculty Group meetings Page 63

FIGURES

Figure 1: Trainees perceptions of the balance of their time Page 27Figure 2: Learning from Unsupervised Work Page 36


6/69

1

Evaluation of the Introduction of the Intercollegiate Surgical

Curriculum Programme

Professor Michael Eraut, University of Sussex

Summary

1. External factors affecting the introduction of ISCP (pp9-15)

The biggest challenge for this evaluation was that the changes introduced by ISCP

were almost swamped by other changes already in progress.

TheEuropean Working Time Directive (EWTD) will reach its final figure of 48 hours a

week in 2009, and this has significantly reduced the time that trainees can spend in both

formal and informal learning environments. This situation has been further exacerbatedby the growth of Sub-specialties, which limit the range of consultants domain of

expertise. The combination of both these factors has led to the dissolution of the firm

structure and the introduction of shift working for both trainers and trainees in several

branches of medicine. In surgery, the overall work patterns of trainers and trainees in a

shift system cannot be matched; so the time they can be in the same place together has

been significantly reduced.

Modernising Medical Careers (MMC) was intended to reduce the time taken to become aconsultant by 2 years, which might have been possible if other factors had stayed the

same. However, the understandable focus on reducing waiting lists for elective surgery

has led to the disappearance of training lists from many hospitals to improve their

productivity. Moreover, the recruitment of the first cohort of the new MMC surgical

trainees through MTAS coincided with the introduction of ISCP and created a very

negative mood among surgical educators, which did not help evaluators seeking data onthe progress of ISCP.

Although ISCP was able to negotiate the restoration of the two lost years, the effect of all

the other changes was a steady reduction of access to training at all levels before the

introduction of ISCP. For example, an increasing number of surgical registrars feelunready to take CCST, because they have had insufficient access to practice during their

postgraduate training.

2.The Evidence Base for this report (pp7-8)

Data collected by the ISCP website, the centre for formal communicationsbetween trainers and trainees.

Three surveys from different agencies: a JCST Quality Assurance survey inNovember 2007, a PMETB survey at the beginning of 2008, and a small survey ofa group of London trainees doing Anatomy courses at RCS.


7/69

2

Four research studies commissioned by ISCP from different deaneries:1. Warwick focussed on continuity of care and factors impacting on surgical

training and patient outcomes. Their evidence was collected from 12 focus

groups between October 2007 and May 2008.

2. Kent, Surrey, & Sussex (KSS) focussed on the nature and value of thesupport for trainers, especially the role of Trust-based faculty groups fromJanuary 2007 to March 2008.

3. The WalesPostgraduate Deanery focussed on the impact of the ISCPwebsite and the assessments it carried from October 2007 to March 2008

4. Southampton focussed on the most critical features for a good placementbetween October 2006 and October 2007.

Six elite interviews with senior surgeons.3. Early Implementation of the ISCP (pp16-26)

ISCP brought in three new features to improve the focus and management of the learning

support system. The first change was a new role, that of an Assigned Education

Supervisor (AES), whose first job was to negotiate the second new feature, that of a

Learning Agreements (LAs) between the AES and their assigned trainees. The third

innovation was the development of a learner led Website, which held the curriculum and

through which communications were expected to be conducted. The website design was

being improved throughout the year, and trainees were more able to use it than some

trainers.

Although most trainers and nearly all trainees appreciated the goals of ISCP, itsimplementation was challenging. In May 2008 78%of validated trainees had an AES and

51% had a Learning Agreements; and there were large variations across both deaneries

and specialties. This makes it very difficult to interpret the data. The evidence reported by

the PMETB survey in February 2008 suggested that AES and LA functions were being

pursued off the website by a substantial number of trainees. 95% of general surgeons said

they had an educational supervisor, who was being responsible for their appraisal, and

78% said they had a learning agreement. This was confirmed by the Wales study; but the

actual use of the LA may have been more limited, because two questions in Table 2 (page17) showed that formal meetings with a supervisor to discuss their progress and formal

assessments of their performance in their current post had not yet occurred for almost halfthe sample.

The evidence on issues relevant to supervision appears to be reasonable, but our small

survey of London trainees in May 2008 discovered a huge variation in the hours spent

working with more senior colleagues. Their estimates for an average week were that 31%spent from 0 to 12 hours, 31% spent from 13 to 24 hours and 38% spent 25 hours or

more. With this type of distribution the use of averages can be profoundly misleading,and the JCST use of dissatisfaction indicators becomes very important.


8/69

3

Table 7 summarises data from the same London survey, which looks at the main sources

of support that trainees recognise and their relative frequencies. SpRs come first,followed by other consultant surgeons, then educational supervisors. However all seven

categories make significant contributions to a significant number of trainees.

Table 7: From whom do trainees get their support, and how much?

Frequency

Rows add to 100%A lot Quite a

bit

A little None No reply

AES 16 32 42 9 1

Other consultant surgeons 14 47 32 4 3

Staff grade surgeons 14 20 32 17 8

SpRs 38 40 17 4 1

Surgeon peers 14 27 44 9 5

Other consultants 6 17 39 20 9

Other health professionals 4 22 36 25 13

The Quality Assurance survey in November 2007 suggests that the issues causingtrainees the greatest concern at that time were as follows:

No induction to Training Programme

Scheduling meetings with their AES

Their Supervisors familiarity with ISCP curriculum

Guidance on Personal Development Planning

Deanery response to their concerns

Trust response to their concernsRota/shift patterns

Service demands of their post

4. Learning Contexts and Access to Learning (pp 27-40)

Table 11: Distribution of trainee time across patient contexts

Ward % Clinic % Theatre % Other %

ST1/FTSTA1

MeanStandard deviation 47.130.3 15.714.0 26.710.5 10.511.8

ST2/FTSTA2

Mean

Standard deviation

47.1

27.5

14.5

13.7

30.1

16.8

8.3

16.5

This table from the London survey illustrates both the distribution of junior trainees time

and the very large variations in those distributions in May 2008. Another question

revealed that, in their view, only ward work was allocated too much time, and theatre and

clinics received too little. 5% spent no time in theatre and 18% no time in clinics. In


9/69

4

addition to time spent and the balance between different settings, the qualitative data

brings out other important factors: the quality of relationships in any particular setting,the appropriateness of the work allocated, the quality and timing of advice and feedback

(undermining feedback is still quite common), opportunities for enhancing theirunderstanding of surgery and sustaining a sense of purpose and progression.

Experienced surgical trainers, both in interviews and informally, also refer to the main

problems in these settings.Elective surgery gets most attention from ST3 onwards, but is

retarded by the lack of trainee access to operational experience and the loss of training

lists to meet urgent targets. Even under the old system trainees are increasingly unready

to become consultants at the usual time. Emergency Surgery and Trauma are allocated

considerable time from higher level trainees, but without the support they need to learnhow best to handle the complex and time critical cases they receive. This is a much

neglected problem in need of urgent attention. Clinics no longer allow time for trainers

and trainees to see patients together, although they can consult with each other aboutpatients examined by the trainee. One consequence is that FRCS examiners are finding

that trainees are becoming increasingly weak diagnosticians. Although ward work take upa great deal of the time of junior trainees, the approach to their learning is surprisingly

laissez faire. The answer to the current difference of opinion between trainers and

trainees over the value of ward work must surely be that some of it is acknowledged as

valuable by both groups, some of it would probably be acknowledged as having littlelearning value by both groups and some of it could be made valuable by giving

appropriate advice and support. This needs to be explored on a wider scale, rather than

leave it to every individual trainer to work it out for themselves. Hence the chapter

concludes with sub-sections on When is Service Work a Learning Opportunity? and

Apprenticeship and Coaching.

5. Competence and Assessment (pp 41-45)

The current assessment advice is both impractical and confused. It neglects the time

required and the difficulty in finding assessors; and it assumes that trainees will suddenly

treat what looks like a test as being formative rather than summative, even when theyhave been reared in a culture of competition. Given the great variation in posts and

circumstances and the ISCP claim to be competence based, it makes no sense for the

three main assessment instruments for junior trainees to be normative rather than criterion

based like the PBA. Nor will most of the available assessors have sufficient experience to

make normative judgements in rapidly changing contexts, a new MMC trajectory and anew surgical curriculum. Trust-backed processes are required that integrate assessments

with the individual trainees ongoing learning and supervision.

6. Teamwork and Relationships (pp 46-50)

There is an increasing recognition of the importance of teamwork both within and across

professions, just as the shift systems are making it more difficult. The key issues are

continuity of patient care and reduction of risk, and improved modes of communicationneed to be turned into communicative practices. It is now very clear that the same issue is


10/69

5

affecting trainees; it is not only patient handovers that need regular attention, but also

trainee handovers for one trainer to another. While there are many positive accounts ofteamwork, these often involve either additional effort which cannot last for long or

limitations caused by rapid changes in team membership (and exacerbated by the shorter4 month placements). This raises the question of whether more attention should be given

to developing different organisational practices rather than expecting individuals to

continuously adjust to frantic changes within the status quo.

7. Training and Support for Trainers (pp51-54)

Much work has been done in the last decade on provision of training and support for

trainers, but the role of trainers has become increasingly difficult; and there are endlesscomments about the lack of time in job plans for teaching. Generally, it seems that

consultants are expected to teach during the time allocated for non-clinical work, but

consultants who take a major role in teaching often have no more time allocated thanthose who take little or no role. Some key individuals in Trusts have started to discuss the

idea of withdrawing funding from those not teaching. Whilst many felt that everyoneshould be involved, they also recognised that some of those who did not wish to make a

significant educational contribution might be poor teachers. Generally, there was much

cynicism from surgeons about getting any support from trust management for the new

educational roles, and this was acknowledged as a problem in the ISCP Pilot workshops.There was no evidence of any financial audit of the manpower gain received by Trusts

through the work done by surgical trainees or indeed of the clinical governance

implications for the Trust of not taking the quality of training seriously.

8. Distributed Apprenticeship and the Organisational Dimension (pp55-58)

There appear to be four possible areas of response to this challenging range of problems,and all of them are important:

1) A risk analysis of the current situation to increase the collective understanding ofall the stakeholdersof the impact on training and service

2) Piloting new approaches to the organisation of surgical training within hospitals,with appropriate backing from the NHS

3) Training of individual surgical trainers4) Training of surgical teams in all the settings discussed above

The main conclusion arising from the evidence collected for this evaluation is thatsurgical education cannot achieve its current goals without significant changes in its

current state. There has been a major reduction in the training time per annum of both

trainers and trainees, and opportunities for trainees and their main trainers to meettogether have been drastically reduced by the new shift systems. The results are that:

1) Trainees for elective surgery will remain safe but fail to reach CCT at theexpected time

2) Training in clinics has become problematic, because joint outpatient lists havecompletely disappeared in most Trusts


11/69

6

3) Training in trauma has been virtually non-existent, and is far from meeting anacceptable standard.

Unless the organisation and funding of training is properly planned, surgical educationwill decline in quantity faster than any conceivable improvements in quality.

The example of The Royal London Trusts reorganisation of their Trauma service shows

that it is both possible and extremely important to conduct ongoing risk analyses of

surgical activities, and to use them for learning by all those involved; because this

provides a crucial direct link between patient outcomes and educational provision. When

connected with the concerns about surgical education revealed by the evidence gathered

for this report, a second conclusion also emerges: that if the current organisationalpractices affecting surgical education cannot implement the changes required for

improving patient safety and other patient outcomes, then the structure of surgical

education will have to be reorganised.

The other major issue is the training of surgical trainers and surgical teams. In addition to

the problem of continuity between team members from different professions, there is arapidly increasing problem of continuity between surgical trainees and their trainers. The

key question to be addressed is that of how far it is possible for surgical trainers to

develop continuity of training for their trainees. Not only are trainers meeting theirtrainees less often, but they know very little about what their trainees may have done with

other consultant colleagues between their own meetings. The first problem is to decide

when communication between trainees and their consultant trainees is needed for

sustaining trainees continuity of learning; and the second problem is that of developing a

meaningful discourse for the mutual understanding of those communications.

This ambitious but very important endeavour could be supported by mediating artefactssuch as recordings or still pictures around which meaningful discussions could take place.For example, still pictures taken at intervals would enable those present at an operation

(not necessarily only surgeons) to add separate short commentaries on each picture about

what they were thinking about at the time they were taken and their later, more reflective,

thoughts. These commentaries do not have to be accurate representations of on-the-spot

thinking, that would be impossible; but subsequent discussions of these commentaries

should help to improve communication and mutual understanding between those

concerned about their respective views of surgical events.

The trust engendered by these initial joint activities should create the interpersonal

relationships needed for addressing the development of teams who can begin to

collectively improve the quality of their service to patients and trainees. Over time this

should help to develop the common discourse, which will be needed if surgical training is

to progress from its original apprenticeship system to a more transparent and reflective

system ofdistributed apprenticeship, in which a group of trainers supports one or twotrainees and offers them the continuity of training that now appears to be essential for

making progress in the next few years.


12/69

7

Introduction

In August 2007, the first cohort to undertake a new Inter-collegiate Surgical Curriculum

Programme (ISCP) began their voyage. This followed 5 years in Medical School and twoFoundation years, in which they were able to engage with six 4 month placements in

different medical environments. Work on this new curriculum, which owed much of its

conceptual framework to the Canadian CanMEDS programme, started in 2002. Two

years of development were followed by a Pre-Pilot year, during which the prototype

curriculum was explored with current trainers and trainees, and the design of a web-based

delivery system was initiated. Then a 30 month Pilot programme from April 2005 to

August 2007 (1) reached agreement on further detail with the Specialty Advisory

Committees (SACs) and the ISCP Assessment Committee, (2) received the approval ofPMETB, and (3) began to prepare surgeons for its formal implementation.

My evaluation role in the Pilot Phase was rather different from that in the Pre-Pilot Phase(Eraut 2005). My time was still very limited, and I had to rely rather more on evidence

gathered by members and employees of the Colleges. The meetings in the Pilot Phasewere concerned only with implementation issues and did not collect any evaluation data.

However, some funds for data collection were allocated to the ISCP Evaluation Task

Group, of which I was a member; and I both contributed to and endorsed its plans. The

data sources used in this evaluation are listed below.

Data sources for the Pilot Evaluation

1. Website data indicating the number and type of website registrations inDecember 2007 and May 2008, and the use of the web facilities essential forthose with special roles.

2. Quantitative data from an ISCP Quality Assurance (QA) survey investigatingnew trainee attitudes in November 2007 (the fourth and last month of their first

ST1 posts) toward a range of ISCP features intended to support their learning.

This survey provides useful data on the learning of ST1 trainees that suggestswhere improvements are needed. However, it was not intended to be part of an

evaluation of ISCP and does not have any baseline data that would enable

comparisons that might indicate whether particular aspects of surgical training

had improved or deteriorated. Nor could it attribute outcomes to changes beyond

the new use of a website, a new support role and the use of some new assessmentsto ISCP, rather than other factors. The survey was based on 350 responses (89%),

and only specialties with more than 10 trainees have been used in this report.

3. Quantitative data from the PMETB website relevant to supervision, feedback,learning environments, critical events and safety, handovers, confidential support

and learning agreements, portfolios and logs. This data was collected over three

months from December 4th

2007 to February 28th

2008, and has been restricted by

the evaluator to General Surgery and Trauma & Orthopaedics (c230 perspecialty).


13/69

8

4. A Trainee Questionnaire in London (n=70) focused on who gives most frequentsupport to trainees, and the perceived balance between service work and training.

5. Four research studies commissioned from different deaneries on key issuesidentified in the pre-pilot evaluation:

Warwick focussed on continuity of care and factors impacting on surgical

training and patient outcomes. Their evidence was collected from 12 focus groups

between October 2007 and May 2008. The participants were 12 consultants, 12

specialist registrars, 10 ST1s, 11 nurses and 6 physiotherapists in the West

Midlands and Severn deaneries. All came from General Surgery or Orthopaedics.

Kent, Surrey, & Sussex(KSS) focussed on the nature and value of the support

for trainers, especially the role of Trust-based faculty groups from January 2007

to March 2008. Their evidence came from documents, including minutes ofmeetings not attended by the researchers; 28 interviews with 23 interviewees from

11 Trusts and personnel from the Deanery and RCS; and attendance at 10 FacultyGroup meetings.

The Wales postgraduate deanery focussed on the impact of the ISCP website

and the assessments it carried. Its evidence came from interviews with 18 traineesfrom different levels and backgrounds and 14 supervisors from a range of

specialties, during the period from November 2007 to March 2008. The trainees

in this sample included ten FTSTA1s and two FTSTA3s.

Southampton focussed on the most critical features for a good placement. Theirevidence came from trainee logbooks between October 2004 and October 2006

(20 from 16 trainees) and interviews with 9 trainers and 6 trainees betweenOctober 2006 and October 2007.

6. Four long and two short interviews conducted by the evaluator with surgeonsknown for their work in surgical education on progression and surgical pedagogy;and one interview with a current trainee with a background in surgical research.

The first three sources were not under the jurisdiction of the ISCP Evaluation Task

Group, but the QA and PMETB surveys may have affected the later decision of the

Evaluation Committee to abandon its planned survey for April 2008, when it becameclear that it would get only a small response rate from its survey weary constituents.

The biggest challenge for this evaluation was that the changes introduced by ISCP had to

be situated in a context where changes in surgical education outside the remit of ISCP

were threatening to swamp the changes being introduced by ISCP itself. Hence, as in the

Pre-Pilot Phase (Eraut 2005), it starts with a review of this rapidly changing context

before discussing the impact of ISCP itself.


14/69

9

Changes in the Surgical Education Context outside ISCP

The Impact of MMC and EWTD

One of the goals of Modernising Medical Careers (MMC) was to reduce the number ofyears it took to progress from Medical School to becoming a Consultant. This replaced

the previous positions of PRHO (Pre-registration House Officer, Senior House Officer

(SHO), and Registrar by two Foundation Years, F1 and F2, followed by six years as a

Specialist Trainee (ST1 to ST6). Although two further trainee years were restored in

2007, the start of ISCP was temporarily eclipsed by the selection process to become the

first group of ST1s. The pioneering F2 cohort had to apply for run through specialist

training jobs in February 2007 in order to get onto a specialist training track in August

2007. This hurdle incorporated several new features and circumstances, whose impactwas not fully anticipated:

1. A much earlier decision on trainees preferred specialty than previously.2. A much higher number of applications for specialist training than previously, for

which F2s had to compete with a significant number of SHOs and, for the firsttime, Non Consultant Career grades (NCCGs) and IMGs (Tooke et al, 2008). This

replaced a gradual reduction in the cohort size by a single selection event, which

rejected a much higher proportion of candidates.

3. A different application system using different criteria for the first time.The first cohort of ST1s started in August 2007, so the successful surgical applicants

became the first cohort of the new ISCP curriculum. However, the new method of

selection was an unmitigated public relations disaster, because of the high, but apparently

unanticipated, number of rejections from a previously untested selection system. Thisdebacle ran in the newspapers throughout the early summer; and its effect on the mood of

surgeons engaged in Deanery meetings preparing for ISCP was very negative indeed.

The loss of time created by MMC might been seen as a possible goal, if surgeons thought

that training could be improved; but the influence of new government targets for elective

surgery was already beginning to diminish the amount of effective training time.Government targets for hospitals have been high on the political agenda for several years,

and its response to very long waiting lists for elective surgery is understandable.

However, one of its side effects has been a reduction in the time spent on training, and

the disappearance of training lists from many hospitals to improve their productivity.

While this target system may eventually change, in theory, the European Working Time

Directive (EWTD) is irreversible. This measure was agreed by the European Union some

time ago; and the maximum number of hours per week for trainees has been gradually

reduced over several years to reach its final figure of 48 hours a week in 2009. This is

very much lower than previous custom and practice, and has significantly reduced the

time that trainees can spend in both formal and informal learning environments. This

situation has been further exacerbated by the growth ofsub-specialties, which limit the

range of consultants domain of expertise. The combination of both these factors has ledto the dissolution of the firm structure and the introduction of shift working for both


15/69

10

trainers and trainees in several branches of medicine. In surgery, the overall work

patterns of trainers and trainees in a shift system cannot be matched; so the time they canbe in the same place together has been significantly reduced. These changes, which

predate the development of ISCP, have been reducing access to training for some time.The Warwick report discusses earlier studies by Barden et al (2002), Bollschweiler

(2001), Gagnon et al, 2006) and Henry et al (2005); and our new evidence has confirmed

that these problems are growing in the following aspects of surgical learning.

Less time for training for both trainees and trainers

This problem developed long before the implementation of ISCP for its first cohort of

ST1 trainees. We collected evidence of this from a wide range of sources, for example a

newly appointed consultant reported a huge reduction in access to practice:

I know from my own personal experience ... at Hospital X ... where I worked for a year...

I had 118 colorectal operating sessions in that year. When the shift pattern came in, justas I left I calculated what my successor would be doing and it was down to 18

operating sessions per year.

A specialist registrar noted limited experience of complex cases:

We are nowhere near as capable as the generation above us in surgery because

technically we have skills. We [may] know how to do something, [but] we just havent got

the exposure to it. Perhaps what well never be as good at is dealing with complications;

we havent had to think on our feet.

An experienced trainer reported how final year registrars no longer felt ready to become

consultants:

Trainees increasingly opt to take a six month to one year fellowship, often overseas, to

get more experience before applying for consultant posts. Over the last few years in one

Deanery (personal experience of 2002 to 2006) approximately 70% of orthopaedic

trainees have been opting to take fellowships, compared with an average of 20% from

other areas of surgery.

This was confirmed by the Head of a School of Surgery at the other end of the country,

who was also considering how best to respond to this problem:

There are a significant, rather frightening, number of people who do have significantproblems in their first 3 years after CCT. The profession, if they think about it, will

introduce mentoring for the first three years in a new job. There would also be some

wisdom in being paired with a more experienced person, who will shepherd you through.

It sounds like a good thing in theory, but I am not sure that it will ever be feasible in

practice.

This problem started at every level at about the same time. For example the same Head of

School reported that:


16/69

11

The ST 2s have one year to do hernias, varicose veins and maybe bowel anastomoses. It

used to be do-able, but now you wouldnt have done more than 20 hernias, 20 veins and

10 bowels: not big numbers. You might do small parts of the first 5, then a bit more then

start to do the whole operation. You wouldnt have done more than 5 complete

operations. He has confirmed this by looking at the trainees logbooks. This exposure is

much less than we think it should be, and theres not much opportunity [to practice],

its very worrying.

The Warwick research group found that all surgical grades perceived a trend towards the

learning of certain skills being deferred to later stages of training, and this compounded

the problem. Thus a consultant reported that people, who had done enough years to be

ready to apply for registrar jobs,had done perhaps 15 appendectomies, while he had doneabout 200 by that stage. This was confirmed by an SHO:

I think its just become more and more expected that ... more is ... deferred to a higheryear in training. I think that the expectation is dont worry about that, youll do it

next year. Thats a recurring phrase.

The view of nearly all surgeons seems to be that patient outcomes are better as there is

more Consultant involvement at both an operative level and ward level:

The patients get a better deal now because theyre getting more Consultant time ... in the

old days you relied on your registrar or senior registrar to run the ward ... (now)....you

do it yourself. I guess in a way patients do benefit from that becausethe most

experienced person ... (is) ... doing the work.

There is also a perception that increased Consultant involvement at an operative level isbeneficial for surgical training:

One positive thing that has come out of this ... is the fact that the Consultant is present in

theatre a hell of a lot more than five years ago and you are taught to do things properly

... we may not be getting the quantity but the quality is better than it was ... (SpR)

The increased involvement of Consultants is however also perceived to be potentially

damaging by SpRs and SHOs, in that there is less opportunity for trainees to act

independently; so trainees will not be sufficiently experienced at the end of their training

to become autonomous Consultants:

I think it will hurt the Consultants in the future because theyll be called in at 3am to do

something that only a few years ago a registrar would be feeling very confident to do

(SHO)

Yeah, its only going to get worse and worse because the more its a Consultant led

service the more theres going to be less of us making decisions. Yes, if we get taken

through it time after time we get to perfect the techniques but if we havent got the

decision making ability, [that will be a serious problem]. (SpR)


17/69

12

Providing continuity of patient care with a more fragmented workforce

This issue was highlighted in my evaluation of the pre-pilot, and was the main focus ofthe Warwick project, who noted that the concept of the patient journey is seen as different

by different groups in different settings. For example, the more senior SHOs, SpRs,nursing staff and consultants consider the whole patient journey, right from first clinic

appointment through to follow-up appointments, whereas the more junior SHOs and

nursing staff are more likely to think of the patient journey in terms of only the period

spent in hospital. This is even more marked for emergency admissions. Surgeons of all

grades perceive that trainees are more involved in the initial stages of the management

and decision making process, but the necessity of handing emergency patients over to the

team on-call means that trainees do not see the whole of the emergency patient journey:

The patients you see in emergency, youre not necessarily going to know whats

happened to them a day, two days a week later... Im sure it used to be that when you

were on call and your team was on call, you knew everyone who came in on call-theywere yours, they stayed yours. But now thats not the case. (SHO)

In the elective setting, more senior trainees and consultants perceive that trainees are

often not involved until the post-operative stages of the patient journey, particularly atSHO level, and that most of the management decisions will come from the consultant.

However, during the post-operative stay in hospital, it is generally perceived by all

surgeons that trainees are able to follow the patient journey more effectively than in the

emergency setting, but the start and end of the pathway are still missing:

Were talking about the routine patients who are booked in for major surgery or routine

surgery, the juniors do tend to follow those patients through, and monitor those patientson the ward and see how theyre getting on. They wont however have seen them in

outpatients prior to them being booked in. And they dont take part in follow up clinics to

see how theyve got on afterwards. (Consultant)

Thus surgeons at all levels and nursing staff all perceive that surgical training is adversely

affected through junior staff not seeing patients at a number of stages.

Theres a pressure now bringing in patients for elective procedures on the day of

surgery and ... there isnt any space or time to see that patient until you join your

Consultant in the operating room who is already with the patient and ready to make the

incision, which isnt the way it used to be ... (SHO)

Moreover, the pre-op assessment clinic is often nurse led; so the post-op ward may be the

first interaction that the (junior) doctor and patient have with each other.

The majority of problems reported relating to patient outcome and not following the

patient journey are due to problems with handovers. Handovers between doctors were

unanimously perceived to be a problem, and for a number of reasons:

With the permanent shift system you can often be handing over three or four times a day

... every handover has the potential to miss off an important aspect, because youve

got one individual trying to get information from another individual and theyre not


18/69

13

always going to get to grips with the whole concept of whats going on with that patient.

If thats happening a couple of times day you can get important things missed or

things done that arent necessary. (SHO)

I think handovers a vulnerable time because I dont think its done with any sort of

system. Its just done ad-hoc and it depends on the people on-call as to how good a

handover they give. And if its very busy then things tend to overrun. Things get missed,

you dont handover crucial details and then a few hours later the ward can contact the

night SHO and theyre quite surprised at the new information. (SHO)

There is no dedicated time in which to formally hand over information to colleagues; and

this new handover culture leads to a diffusion of responsibility amongst trainees, which

itself has knock-on effects for the patient and for training. This is a view held inparticular by consultants:

The whole working ethos has now changed because of the shift system where you feel that

once youve finished your shift, your responsibility is over, you hand over to somebody

else. Its that persons responsibility. Whereas previously, we would have come the next

day to find out what would have happened to that patient. I think that whole ethos has

changed. (Consultant)

This current model fosters a culture of not actively dealing with problems. So if you have

a slightly difficult patient there is a tendency for the juniors not really to actively manage

it but to pass it onto the next team. (Consultant)

Providing continuity of training when trainees and their trainers are only

intermittently scheduled to be in the same place at the same time.

The main response to these changes has been the emergence of a consultant led service,

which supports the governments aim of improving the quality of care. However, this has

now reached the stage where the only continuity of care provided is through the patients

consultant; and this means that there is only a weak back up if the consultant is called to

cope with an emergency elsewhere. The introduction of the shift system and the collapseof the firm have meant that consultants have less continuity of contact with their

trainees. In addition to continuity of care, there is also a need for:

continuity of contact between the trainee and the named trainer because of the

seemingly unavoidable necessity to have shift patterns that arent synchronised. It means

that very often trainers are working with a trainee [with whom they are not very

familiar], which means that nearly every training opportunity has to go back to base one

and start again. (Consultant)

I think consultants tend to be a little bit more protective of their patients than perhaps

they used to be in the past, because theres so many shift changes and often a consultant

will be on call with a Registrar that theyve never met before, [and whom] they

dont know anything about. And there are consultants that are not happy to trust the

Registrar and what they say in terms of yes I can do this. (SpR)

The handover of trainees is yet another handover problem to be negotiated.


19/69

14

Organisational factors

One of the recommendations of the Pre-pilot Phase was that each Deanery should have a

School of Surgery to provide a focus for surgical education in its region. This has nowbeen implemented and contributed greatly to the Pilot Phase and the implementation of

the first year of ISCP, as well as the recruitment of the first ST1 cohort by the Deanery.

This makes it easier to create situations that allow trainees to live in the same place for 2

years by planning rotations across hospitals that are reasonably close.

However, no comparable change has occurred at Trust level. Surgery is strongly

controlled by government targets and outcomes data; but surgical education is left

without any such focus. Hence, one Head of School argued that the target culture, whichmost surgical educators perceived as squeezing out training time in theatre, could only be

resisted by making training itself a target.

This influence of targets on training occurs in three main ways.

1. Pressure on consultants, some of whom are hugely overworked and devote lesstime to trainees:

I think the Government directives to reduce waiting times are a big pressure

because were always getting interrupted when were in the coffee room between

cases, by the waiting list co-ordinator with her big black book saying this

patients about to breach, we need to add this patient on. I see the waiting list co-

ordinator probably once or twice a day with her big black book.

One consultant said that the waiting list pressure was so strong that he was now

working 96 hours a week, twice the new EWTD limit; while another described hisbalance between orthopaedics and trauma as 80% on both. Another factor is that

consultants now have a lot more reporting duties, which take up yet more of their

time, but no longer get any support for the increasing amount of administration.

2. Pressure on training opportunities, with Training Lists becoming quite rare inmany Trusts.

Training lists, i.e. operations for the trainee to do are clearly not possible in the

context of service targets driven by the Department of Health. (Consultant)

Sometimes even small bits of surgery are frowned upon:

Ive been in theatre and Ive had staff tell me you need to get your registrar in to

do this, because it takes too long. I was taking perhaps a couple of minutes

longer to close a laparotomy than the registrar would have done, and I was doing

it on my own. (SHO)

3. The 4 hour A & E target is criticized by surgical trainees and nurses, because it

removes carefully negotiated opportunities to operate and some patients end up in

inappropriate wards before being properly assessed:


20/69

15

I think the biggest impact since weve been here is the amount of moves a patient

has ... (later in focus group) ... continuity of care to me is somebody being

admitted somewhere and hopefully being discharged from there. (Staff nurse)

A similar problem arises when there are split sites. Even when the patients stay in the

same beds, their doctors may move from one site to the other, thus weakening the

continuity of medical care. Adverse effects were recorded and trainees concerned about

not being able to follow their patients:

We have an elective hospital and an emergency hospital ... so Im there for a 12 hour

period, admit a patient with appendicitis, do an operation on them and then I never see

them again, which is frustrating and probably not great for patient care. (SpR)

MMC effects on the organisation of trainingalso contribute to some of these problems.The Warwick study found that the reduction from 3 to 4 years as an SHO to 2 years as a

ST1-2 appears to have created a shortage of junior doctors, some of whose traditional

work now has to be done by someone else. Some of this work is now being handled by

nurses, and this provides a safe consistent service; but it does remove some opportunities

for junior doctor learning, for example in pre-op. More worrying is the tendency for

higher level doctors to be given some of these former SHO jobs at the expense of

activities more appropriate for their role. One SpR described this as acting down, and it

clearly contributes to the slower progress of their training.

The fact that the SHOs arent there is impacting acutely, in that we are acting down, I

may not have an SHO during the day, and at night theres no SHO so I cant go home.

The use of 4 month placements has received almost universal disapproval. One problem

concerns trainees settling in time, and another is the lack of a long enough stretch of

practice to reach a satisfactory level of competence, and having to pick it up elsewhere.

If youre doing an orthopaedic post, you just get around to doing that DHS, youre

learning how to do it and then youre off. So you never get to do that DHS from start

to scratch. Its the same with appendectomy. Coming from the old BST I did six month

jobs. By month three of four I was going solo with the appendices and the DHSs and then

I had another two or three months to build on that experience. And you develop so much

quicker once you get to that certain point but if youre cut down ... you dont even get to

that three month stage.

Trainers have a similar problem, because they need to spend time with new trainees in

order to assess what they can do, whether they can trust them and whether their trainees

self evaluation is adequate for both safe practice and new learning challenges. An ENT

consultant suggested it took six weeks for trainees to find their feet and that a training

gain only occurred after the fourth month when independent operating practice becameviable. Many consultants claimed that four months posts were a major handicap.


21/69

16

Early implementation of the ISCP curriculum

Educational Supervisors and Learning Agreements

ISCP brought in three new features to improve the focus and management of the learningsupport system. The first change was a new role, that of an Assigned Education

Supervisor (AES), whose first job was to negotiate the second new feature, that of a

Learning Agreements (LAs) between the AES and their assigned trainees. The third

innovation was the development of a learner led Website, through which communications

were expected to be conducted.

The general progress of ISCP can be traced through data from the website itself, although

access to personal data is very strictly confined to a limited number of people with theappropriate passwords. The first step for a new trainee is to contact the Programme

Director for their Specialty in their Deanery, who then validates their position. 50% of

new trainees were validated by mid-September 2007 (their second month). This reached75% in December 2007 and 92% in May 2008. Clinical Supervisors had to be approved

for the AES role by the Programme Director, and their first duty in this new role was todevelop a Learning Agreement with each of their trainees. The progress of AES

appointments and agreed LAs is summarised in Table 1 below, which also shows a wide

range of uptake by both deaneries and specialties. This demonstrates that some deaneries

and some specialties have been much more effective than others in implementing thesecrucial aspects of ISCP.

Table 1: % of validated trainees with an AES and a Learning Agreement

Website data % of validated trainees withan Assigned Educational

Supervisor

% of validated trainees withLearning Agreements

December 2007Range of % across

deaneries and

specialties

OverallMean 44%Range of Range of

Deanery Means Specialty Means

24-63 23-54

Overall Mean35%Range of Range of

Deanery Means Specialty Means

8-65 18-51

May 2008Range of % across

deaneries andspecialties


Deanery Means Specialty Means57-96 72-92


Deanery Means Specialty Means30-75 42-58

Not surprisingly, the JCST Quality Assurance survey in November 2007 (using 1-5

scales where 1 expressed greatest dissatisfaction) found that many trainees found it very

difficult to schedule meetings with their AES. Trainees from three of the seven larger

specialties gave more negative than positive ratings for scheduling meetings with their

AES (average 3.0). The most negative response of all concerned establishing andmanaging an online Learning Agreement (2.6, see Table 8). This helps to explain why the

response rate dropped to one third, when the same sample was asked about the extent to

which they were able to reach all the learning objectives in their Learning

Agreement. Moreover, 20 % of this reduced sample felt unable to reach the agreed

standard in 4 or more of the 12 areas of competence.


22/69

17

Unless the sample was very biased, the higher figures reported by the PMETB survey,

which finished at the end of February 2008, suggest that the AES and LA functions werebeing pursued off the website by a substantial number of trainees and trainers. 95% of

general surgeons said they had an educational supervisor, who was being responsible fortheir appraisal, and 78% said they had a learning agreement; and the figures for Trauma

and Orthopaedics were almost identical. This was confirmed by the Cardiff study; but the

actual use of the LA may have been more limited, because two questions in Table 2

below showed that key meetings had not yet occurred for almost half the sample:

Table 2: Meetings about progress and formal assessment in General Surgery

PMETB survey

Have you had a formal

meeting with your supervisor

to discuss your progress?

Have you had a formal

assessment of your

performance in this post?

Yes and it was useful

Yes, but it wasnt useful

No, but this will happen

No, but I would like to

525

30

13

428

37

14

One consultant recognised the teething problems caused by the new AES role, andreported that he and his colleagues would know better next time round.

Now that the ISCP curriculum is explicit, we can see when trainees have not had good

continuity of learning, partly because they have not engaged with their AES. Neither the

AES nor the clinical supervisor understood their roles at that time; but we hope it will be

different this year.

They were accustomed to looking after 2 posts in their hospital, and had embraced theMMC concept of a run-through training programme; so it was a challenge to develop astructure that provided educational supervision for 17 trainees. The guidance from ISCP

is that the AES should not also be a Clinical Supervisor; but he thought it would be better

if each AES became a clinical supervisor for at least one trainee. Otherwise there would

be no incentive to do the AES job.

Nobody is purely altruistic. A lot of people are interested in clinical supervision where

there is direct interaction with trainees and you can see them develop. Nobody is

interested in remote advising, which is very dull and time-consuming, and notintrinsically rewarding.

One advantage of distinguishing the role of AES was that it recognised the need for more

than one trainer in many posts.

The Wales study focused on the new Learning Agreements (LAs) and the website; andtheir data was gathered between November 2007 and March 2008 (months 4-8 of the first

cohort of STs) from a sample of trainees and a sample of Clinical Supervisors, who


23/69

18

supported the trainees on a regular basis both before and after ISCP was launched. Both

parties found these new LAs helpful for guiding their subsequent work.

Unless they were absent from work, most trainees and clinical supervisors attemptedjointly to access the ISCP website in the first weeks of a post. Where the supervisor or

trainee had some facility with the website, learning agreements were generally followed

through as intended and benefits to learning appeared to result. However, widespread

delays in completing the learning agreement were caused by administrative and website-

related difficulties. Some agreements were signed off so late as to render them

ineffectual, and mid-point meetings were often ignored, particularly in 4 month posts.

The small but recent London study in May 2008 showed that about a third of trainees

first discussions with supervisors did not occur in the first four weeks of their posts, andone eighth of them never took place.

Despite these inauspicious beginnings, most trainees and clinical supervisors agreedinformal plans for training posts, recorded them on paper and followed through with

ongoing interactions (while awaiting the removal of obstacles in the system). However,some participants reversion to pre-ISCP approaches to specialty training inevitably

diluted the impact of the ISCP learning agreement

There was a general assumption that trainees should be proactive in preparing for andsetting up the learning agreement. Some trainees carried forward an awareness of their

learning needs from previous posts; others such as ST1s were dependent on a prescription

of generic objectives by the supervisor. The supervisors role at the initial meeting was to

mediate between the trainee and the particular type of post on offer: clarifying learning

needs, setting learning objectives and identifying related practice opportunities in atraining post. Most trainees accepted that the learning agreement was the outcome of

negotiation with their supervisor of global objectives, personal learning objectives andthe realities of the post.

Trainees were predominantly satisfied with the balance of objectives achieved in their

agreements and with the negotiation process. One FTSTA3 trainee managed to agreeobjectives that met his interests in vascular and general surgery, including the kind of

operative experience he was seeking; but another found that her need for advanced

experience in theatres was limited by competition from a specialist vascular registrar. She

also felt that the levels set for her in the ISCP learning agreement had been too low:

The expectation from the programme at my level is just to be able to clinically manage,

recognise patients with problems and know what to do to formulate a management plan,

which I have achieved already. (FTSTA3)

Trainers in the Southampton study felt they needed more formal support to keep up to

date with all the aspects of being an educational supervisor. For example:


24/69

19

There is little training and support on how to deal with a difficult trainee and I need to

develop skills to tell a trainee when they are not doing well, as it is often easier not to do

it.

One trainer in the Wales study, who used the ISCP criteria to highlight gaps in her

trainees core surgical skills and knowledge, was very positive about the new system:

The learning agreement) does focus your mind as a traineras to what is expected of a

trainee at different levels and I think it is very objective, so I am actually enjoying it

because I can feel Right, this is what I have to be doing and make sure that I have at

least arranged for the minimum of that to be examined. (General Surgeon)

However, learning agreements required protected time if they were to provide a

continuing focus for supervisor and trainee. In a service-oriented post, annual leave of

trainer and trainee, on call, night shifts, and a trainer sometimes away from the basehospital caused a mid-point meeting to be held back until the end of the post, when an

Orthopaedic Surgeon felt: it was less than satisfactory than what we had initially set outto do. Two orthopaedic surgeons also admitted to having underestimated the time

required to complete ISCP supervisory tasks; while a general surgeon found it impossible

to set a time: to sit down and reflect, because with their rotas and shift system I never

knew when (the trainee was) going to be around.

The ISCP Website

The implementation of the ISCP was totally dependent on the website, because it was the

central channel for communication between the many different role holders and thetrainees. Indeed, the whole training process became learner-driven through the website,

and could not easily function properly without it. This website was a major focus for theWales research, which identified 24 different problems from their interviews with 18

trainees and 17 from their interviews with 14 trainers; and its prominence in the KSS

research reflected the considerable attention given to it during Faculty Group meetings in

the first quarter of 2008.

Administrative delays and an initial lack of facility with the ISCP website highlighted the

need for a full and timely induction to ISCP. Some trainees were concerned that they had

only received a brief announcement of the inauguration of a web-based system, but those

who had received face-to-face guidance on the website reported an immediate boost totheir confidence and ability to engage with ISCP.

Site design, with 15 trainee problems and 7 trainer problems in the Wales study and a

cumbersome to use verdict in KSS, was the most prominent type of complaint. Several

people compared it unfavourably with other similar sites in terms of its user-friendliness.

I find it difficult to use, and I'm committed. I find it not a pleasurable site to use,

especially at the end of a busy day. (Surgical Tutor, November 07)


25/69

20

There were major difficulties transferring records from logbooks on other sites which

some trainees were using. People complained both about too much unnecessaryinformation and missing information, such as insufficient sub-specialty-related

assessments. The Wales study uncovered 7 types of wrong information, and KSS founddifficulties in correcting errors. There were also concerns about confidentiality, losing

data (which did occur) and possible undesirable uses of the facility such as unthinking

tick boxing. Other problems included difficulties in registering (including not receiving a

password even when individuals had pre-registered), problems accessing the site from

various computers and relevant people not having access to the site. More seriously still,

some Welsh Trusts blocked websites and three of our four commissioned studies reported

that some consultants refused to use them. One SpR, for example, explained why he had

had to change his supervisor:

If youre a supervisor you have to be logged on ... and there are many Consultants ... who

are just flatly refusing to go onto the system. Therefore, they [cannot be an] officialsupervisor; so people are going elsewhere to get their supervisors I wrote to the ISCP

and my programme director requesting a change [of supervisor] because there was no

way that he was ever going to get online and sort it out ... I changed my supervisor to

another Consultant and weve been online doing all the things properly; but there are

many Consultants who cant be bothered with it.

Many of the problems were relatively minor, but nevertheless time consuming and

irksome to people who were not in any case convinced of its benefit. The concept of a

web site being a central part of the curriculum was very new to everyone. In general,

trainees were more accepting of the idea than consultants and liked having an explicit

curriculum. They were generally quite positive (but not to the extent of beingenthusiastic) about having a record of what they'd achieved as they felt it was a moretransparent system. In KSS there appeared to be a division between STs/FTSTAs who

had come through the Foundation Programme and those who hadn't. The former group

were familiar and comfortable with the concept of a web-based system, and sawadvantages in it. The latter group found it more of a learning curve and were more pre-

occupied with the practical aspects.

Many trainees effectively taught their consultants how to use the site, or organised it so

consultants had minimal engagement. Some trainees and surgical tutors encountered

problems persuading educational supervisors to use the site and felt that there had been

insufficient training. However in some trusts only a minority of consultants attended thetraining that was provided. It was noticeable that at both the mock-ARCP

1meetings

observed, the majority of trainees had few or no assessments recorded on line, despite

advance notice of the meetings.

By the end of the research, both groups were only beginning to understand how the web

site worked, and it is therefore too early to assess its value.

The juniors are taking it up more and more, consultants less so but getting better.

(Surgical Tutor)

1The Annual Review of Competence Progression which replaced RITA


26/69

21

The Wales study concluded that continuing feedback from users had been sought and

acted upon to overcome administrative and website-related delays, and to refine a rigid,over-laden website; and recommended that:

Efforts to ease navigation difficulties and improve the flexibility of the ISCP website

should continue, and a full and early induction of new entrants to ISCP should be

provided.

Initial response to ISCPThe teething problems of the website notwithstanding, many trainees in the Cardiff study

could appreciate the continuity between ISCP and the Foundation; and experienced

trainees, comparing ISCP with the previous system, recognised its potential to improve

their learning:

What used to happen is that you could fly along and not be sort of reviewed or assessedin any formal way and you come to the end of your 6 months in the job and sit down and

have a relatively informal sort of assessment. And as long as you kept your head above

water, then youd be fine to progress to the next stage. With (ISCP) in place, it provides

a more formalised, a more structured approach and I do feel that when all participants

are embracing it, then it works very well. (BST3)

For most trainees, the ISCP clarified what they could expect in a post and gave

milestones to measure progress over time, thereby reinforcing their growing sense of

confidence and efficacy in conducting procedures. However, one trainee found the ISCP

curriculum to be something of a mystery:

There is definitely a curriculum for Foundation programme. I dont know if there is for

the specialty training.I suspect its buried in the ISCP website under the general

objectives. They want to produce doctors who are competent at acute management of

the patient. That keeps coming up time and time again. But we dont all get exposure to

those things. I know that in the run-through medical training they have to be assessed on

three central line insertions. They never do central line insertions, but its in the list to

have been done. (FTSTA1)

Trainees recognisedthat their access to operative experiences was determined by factors

in posts that ISCP could do nothing about:

Theres been a job where theres just been the consultant and me in the hospital and Ive

done lots and lots of things myself. Whereas when youve got a lot of other junior

doctors and various degrees of seniority between you and a consultant, you tend to end

up being a house officer and not getting to do that much. (FTSTA1)

The JCST Quality Assurance survey of November 2007 provides a useful picture of the

first few months of ISCP and three sections of it are presented and discussed in this sub-

section: Supervision, Feedback and Support, and Organisational Issues. These will be

accompanied by some evidence and a smaller London survey in May 2008. Table 3


27/69

22

below presents the QA data on supervision, and is followed by some later data from the

PMETB survey in February 2008.

Table 3: QA survey data on Supervision (1-5 scale)

Issue Mean % 1 or 2 Specialties below 3

Educational supervision

Clinical supervisionOut of hours experience

Elective experience

Emergency experience

Consultant led ward roundsGaining consent for procedures

3.5

3.83.6

3.6

3.5

3.63.9

20.7

11.614.4

16.9

18.9

24.024.0

One 2.8

One 2.7

Whereas clinical supervision was a familiar role, the concept ofeducational supervision

was more recent in origin and had not previously been defined. It only became formalised

with the introduction of the role of an AES and the use of Learning Agreements; so it isnot surprising that it gave rise to a higher level of dissatisfaction than the familiar role of

clinical supervision. However, the differences between General Surgery and Trauma &

Orthopaedics, confirmed by the later data from PMETB in Table 4 indicate that other

factors might also be involved in accessing clinical supervision that are absent from

informal feedback.

Table 4: PMETB data on clinical supervision, informal feedback and ethics

Do you know who provides clinical supervision when you are working?

General Surgery Trauma & Ortho

Yes and they are accessible 81 67

Yes, but they are not easy to access 8 18

No, but there is usually someone I can contact 8 12

No, I have been left without help at times 2 4

Frequency of informal feedback from a senior clinician

Daily 4 4Weekly 24 22

Monthly 34 33

Rarely 30 33

Never 8 10

There are simple reasons why T&O clinicians might be less accessible, and this does not affect

the pattern of informal feedback from senior clinicians. However, the amount of informal

feedback appears to be remarkably small. This was confirmed by the Wales study, which found

that preoccupation with work in the wards starved some trainees of operative experience

and was a common cause of dissatisfaction. Other than in the initial meeting, they

obtained very little contact with their supervisors during their posts and little or no

feedback from their seniors. Factors such as a shortage of SHOs, an overstretched


28/69

23

consultant and competition from a surgical assistant were cited as contributing to

unfavourable experiences in training posts. These trainees still accessed the ISCPwebsite and defined learning objectives to enhance their self-directed learning, but their

isolation gave rise to undue caution and diminished ambition in specialty training:

I think for effective learning you shouldnt expect too much, you shouldnt exert yourself

too much. You should take things that are within your grasp. (If you) expect too much

from that four months. I think you will have a nervous breakdown. (FTSTA1)

Taking a particular interpretation of self directed, some trainees were inclined to blame

themselves if a training post fell short of their expectations. Very few identifiedinadequacies in the Trust organisation and teaching: the formal teaching sessions, the use

made of X-ray meetings and post-take ward rounds and the follow through on the care of

individual patients. We also need to consider the huge variation in trainee experiences.

The small London study for example, based on trainees attending anatomy courses atRCS, discovered a huge variation in the hours spent working with more senior

colleagues. Their estimates for an average week were as follows:

31% spent from 0 to 12 hours31% spent from 13 to 24 hours

38% spent 25 hours or more

With this type of distribution the use of averages can be profoundly misleading, and the

JCST use of dissatisfaction indicators becomes very important. Table 5 below gives some

of the QA data on appraisal and feedback.

Table 5: QA data on Appraisal and Feedback (scale 1-5)


Fairness of appraisal process

Frequency of appraisal process

Quality of feedback received

3.6

3.3

3.5

7.8

17.4

11.4

One 2.9

The small London survey in May 2008 asked about the quality of feedback rather

differently. Respondents were asked to note their position on a continuum from 100%

constructive feedback to 100% undermining feedback. Although the balance is generally

positive, only 75% of trainees received more constructive comments than underminingcomments. This difference could be due to the small size of the London sample, the later

timing of the survey or a combination of these two factors.

Table 6: The balance between constructive and undermining feedback

% feedback

constructive

100 80/90 60/70 50 30/40 20 10

12 42 21 9 5 10 1


29/69

24

Table 7 below summarises data from another London finding, which looks at all the

sources of support that trainees recognise and their relative frequencies. SPRs come first,followed by other consultant surgeons, then educational supervisors. However all seven

categories make significant contributions to a significant number of trainees.

Table 7: From whom do trainees get their support, and how much?

Frequency

Rows add to 100%A lot Quite a

bit

A little None No reply

AES 16 32 42 9 1

Other consultant surgeons 14 47 32 4 3

Staff grade surgeons 14 20 32 17 8

SpRs 38 40 17 4 1

Surgeon peers 14 27 44 9 5

Other consultants 6 17 39 20 9

Other health professionals 4 22 36 25 13

On this the QA data showed that a sixth of the trainees were also concerned about theirsupervisors familiarity with the ISCP curriculum and over a quarter doubted their abilityto provide guidance on the Personal Development Planning which underpinned it.

Table 8: QA data (1-5 scale) on Induction


Induction to Hospital/TrustInduction to Department

Induction to Training Program

Scheduling meetings with AES

Establishing & managing an

online Learning Agreement

Supervisor familiarity withISCP curriculum

Guidance on Personal

Development Planning

3.53.7

2.6

2.8

2.6

3.3

2.8

0.82

17.4

3

31.1

24.3

30.6

16.4

28.7

All 2.5-2.9

One 2.7 Two 2.8

All 2.4-2.9

Two 2.9

One 2.5, One 2.7,

Two 2.9

Trainees tendency to blame themselves for all their problems may have also beenconfirmed (1) by the findings of the PMETB survey that 45% of trainees had not beentold who to talk to in confidence if they had personal or educational concerns and/or (2)

by the difficulties encountered by up to a quarter of them when they tried to seek help

from their Deanery or their Trust, the two organisations responsible for their well being

(see Table 9 below). Neither was particularly responsive and their trainers also felt that

there was little they could do about decisions and practices at an organisational level,

often because of the wider issues raised in the previous chapter. However, this should not

2Yes/no question

3Yes/no question


30/69

25

become an excuse for just ignoring individual concerns, especially in relation to rotas or

shift patterns (26.8% dissatisfied in Table 10 below) and the service demands of theirpost (22.4 %). For example a trainer, who contrasted the relative enthusiasm of two

trainees, also admitted that a shortage of ST1s and ST2s created a service workload thatdeflected his trainee from making the most of his opportunities.

Table 9: QA data (1-5 scale) on Response to Trainees Concerns


Opportunities to provide

feedback on Training

Confidentiality of feedback

Deanery response to concerns

Trust response to concerns

3.5

3.3

3.0

2.9

20.5

9.8

19.2

25.0

One 2.9

One 2.9

One 2.8, Three 2.9

Three 2.7, One 2.8

Table 10: QA data (1-5 scale) on Organisational Issues


Rota/shift patterns

Administrative Support

Access to IT & Internet near

ward/theatreAccess to Mandatory Courses

Service Demands of Post

3.2

3.5

3.6

3.4

3.3

26.8

16.9

16.9

16.2

22.4 One 2.9

Another consultant said that ISCP improved the professional dialogue between

supervisors and trainees by encouraging focused discussion of cases:

A chap came in and was diagnosed renal colic, but he was youngish and had a ruptured

aneurism. That actually focused a very good training session, to sit down and actually

talk about it, getting the trainees point of view as to how he managed the case and theproblems that he had. (General Surgeon)

This differed from his former practice of simply telling the trainee to think about ruptured

aneurisms in connection with renal colic.

This type of case discussion can readily promote unplanned and reflective learning by

trainees, for example, on issues arising from a complication. A general surgeon noted:


31/69

26

My ST3 missed a couple of X-ray diagnoses in a short period of time that were quite

important and quite serious, and he realised that. I realised hed missed one of them but I

didnt realise it was two, so we talked about it and discussed it. So it was reflection from

his point of view. He realised hed missed something as hed been to an audit meeting

where names are not mentioned but people are criticised and (they) say How could you

miss that? We talked about it and arranged for one of our radiology colleagues to meet

with the ST3. (General Surgeon)

This cons