Audit of Consultation Good Management Practice: guidance for All

General Medical Council (2011)

______________________________________________________________________________

Audit of Consultation on

Good Management Practice: guidance for All Doctors. Raising and Acting on Concerns about Patient Safety, Writing References

Final report

28th October 2011

www.srcentre.co.uk Page 1 Final Report


______________________________________________________________________________ www.srcentre.co.uk Page 2 Final Report

ACKNOWLEDGEMENT

The Social Research Centre wishes to thank the

General Medical Council’s Standards and Ethics Analysis Team for their helpfulness during our conduct of this audit.



DISCLAIMER

This report has been prepared for and only for the General Medical Council (GMC) in accordance with the terms of reference specified to Social Research Centre’s (SRC) in 30th August 2011 and for no other purpose. The opinions expressed by the participants in this consultation are strictly those of the person who gave them and not SRC. The Social Research Centre does not accept or assume any liability or duty of care for any other purpose or to any other person to whom this report is shown or into whose hands it may come save where expressly agreed by our prior consent in writing. For convenience, this document may have been made available in electronic as well as hard copy format. Multiple copies and versions of this report may therefore exist in different media. Only the final hard copy should be regarded as definitive.



CONTENTS

Section Page No 1 EXECUTIVE SUMMARY 5 2 INTRODUCTION AND BACKGROUND 7 3 WHAT WE DID 11 4 WHAT WE FOUND 16



1. EXECUTIVE SUMMARY 1.1 BACKGROUND TO THE AUDIT

This report is an independent audit, prepared by the Social Research Centre (SRC) (ww.srcentre.co.uk) of the consultation data submitted and gathered in relation to the General Medical Council’s (GMC) draft guidance, “Good Management Practice: guidance for all doctors; Raising and acting on concerns about patient safety; and Writing references”.

1.2 TERMS OF REFERENCE In essence, the audit was focused on: Checking the degree of compliance with the instructions that GMC’s Analysis

Team had specified for data preparation and analysis; and, Assessing the extent to which the subsequent interpretation of the data was fair

and reasonable.

1.3 WHAT WE DID

SRC agreed the priority data sets for examination with GMC and agreed a sampling and audit approach that tested the extent to which the data, findings, conclusions, recommendations generated by SRC concurred with those generated by the GMC Analysis Team in terms of completeness, accuracy and substance.

1.4 WHAT WE FOUND

Overall, the audit had found that the GMC Analysis Team’s approach to analysis

and interpretation was both thorough and compliant with the instructions given.

There were very high levels of completeness and accuracy throughout the data sets examined with virtually all of the substantive points included.

In so far as SRC can judge, the correct interpretation has been applied to all of the

points raised.

We have listed a number of general points which GMC may wish to reflect upon for regarding the analysis of data in future consultation exercises.



Summary of Audit Results

Data Set Audit Result Hard Copy Questionnaire Responses That Were Entered Manually

Very high level of completeness and accuracy.

‘Open ended’ written submissions

Very high level of completeness and accuracy - Average of 98% on both aspects.

We would stress that the specific aspects that created an apparent shor tfall short fall below 100% are not material.

Meeting Notes

All substanexamined

tive points were identified in the sample

Audit of Individual Questions

antive points were identified in the sample examined Virtually all of subst

We would stress that the number of specific aspects of different are minimal and not material.


______________________________________________________________________________

2. INTRODUCTION AND BACKGROUND1

2.1 BACKGROUND TO THE CONSULTATION 2.1.1 Background to the Development of the Guidance

The GMC’s 2009-10 work plan included a review of Management for Doctors (2005). The Standards and Ethics Committee agreed that the review of Management for Doctors should be taken as an opportunity to develop guidance of more general interest, assistance and relevance to doctors who do not necessarily have a formal management role.

It was also agreed to review at the same time two pieces of supplementary guidance Raising concerns about patient safety (2006) and Writing references (2008). See www.gmc-uk.org/guidance/ethical_guidance.asp

The Standards and Ethics Committee agreed that a Working Group should be set up to take forward the review of the guidance. The Working Group, chaired by Dame Joan Higgins, met four times between April 2010 and January 2011 to further develop and agree the content of the new draft guidance. At its meeting in January 2011, the Working Group approved the draft guidance subject to some further drafting changes. The three pieces of draft guidance were entitled, “Good Management Practice: guidance for all doctors; Raising and acting on concerns about patient safety; and Writing references”.

2.2 THE CONSULTATION PROCESS 2.2.1 Launch

The consultation on Good Management Practice: guidance for all doctors and the supplementary guidance Raising and acting on concerns about patient safety and Writing References was launched on 4 March 2011 (following Council approval on 17 February) and closed on 3 June 2011. Requests for extensions for submitting responses were granted to some organisations, including the British Medical Association (BMA), on request.

1 The description of the background to, the operation of, the responses from and the analysis of the consultation process was provided by GMC.

www.srcentre.co.uk Page 7 Final Report

http://www.gmc-uk.org/guidance/ethical_guidance.asp



2.2.2 Consultation Process

The consultation consisted of a long and short questionnaire.

The long questionnaire asked 35 questions on the draft guidance (including on the supplementary guidance Raising and acting on concerns and Writing references); it included detailed drafting questions and questions on specific issues that had arisen during the review, as well as questions about the clarity and level of detail of the guidance. Respondents had the opportunity to comment on each section of the guidance, even where we had not identified a particular issue or question.

The short questionnaire was targeted at individual doctors. It asked 8 main questions (some with sub questions) about the key issues and did not require respondents to have read the draft guidance.

The consultation was sent to over 600 key interest groups and individuals across the UK. A press release was issued which was picked up by a number of media outlets and reproduced on a number of websites.

Information about the consultation was also cascaded through the Department of Health (England) Medical Directors Bulletin and the GP and Practice Team Bulletin.

A double-sided postcard targeted primarily at patients/public was produced which advertised both the management and prescribing consultations. This was taken to events and meetings and was also distributed through Bingleys List to 395 Patient and Public Involvement Leads at NHS Trusts and Health Boards across the UK as well as some national organisations. We heard back from a number of these organisations seeking copies of the consultation documents, and others advised that they had cascaded the information through their networks or added it to their websites or facebook pages.

Reminder emails and letters were sent to contacts 2 weeks before the end of the consultation encouraging them to respond.

2.2.3 Consultation Responses

Questionnaire

Respondents were encouraged to respond online through the GMC’s consultation website ‘Community People’, but were also provided hard copy consultation documents on request. Some respondents did not complete the questionnaire but provided comments on particular sections of the draft guidance, or issues of particular interest to them. All hard copy and email responses received within a week of the consultation closing date were inputted into the consultation database so the data could be analysed together.

Where respondents requested a longer extension, GMC advised that due to time pressures, that it would run the consultation reports for everyone else and start the analysis before it had received their responses. Consequently, these responses were



not entered into the database and therefore were not counted in the quantitative analysis. However, the GMC assured these organisations that the comments they had made would be taken into account as part of the qualitative analysis.

GMC received 208 responses to the consultation, made up of 116 responses to the long questionnaire and 92 responses to the short questionnaire.

Consultation events/meetings

Throughout the consultation period, the GMC’s Standards and Ethics Analysis Team (Hereafter referred to as the ‘GMC Analysis Team’) attended a number of meetings and held a small number of specific events to seek views on the draft guidance and to encourage people to respond to the written consultations.

2.2.4 Contextual and Operational Constraints upon the Consultation Process

The consultation fell at a time when the GMC was undertaking other, high profile, consultations including the initial scoping exercise on Good Medical Practice and consultations on reforms to its fitness to practise procedures and adjudication. In consultation with colleagues in its Strategy and Communications Directorate, the GMC explored the best ways to promote the guidance without adding to consultation fatigue, particularly when approaching patient and public groups.

As a result, GMC relied heavily on cascading information about the consultations through existing networks and bulletins as well as seeking the opportunity to attend established meetings of other groups and conferences, rather than to hold events of its own.

2.2.5 Analysis of the Responses to the Consultation

All of the data collected throughout the consultation process from the written questionnaires, and notes of the various consultation meetings and events were collated and analysed by the GMC Analysis Team between June and July 2011, following an established procedure and instructions for analysis.

Reports were prepared, including recommendations for drafting changes, for consideration by the Working Group. The Working Group met twice since the close of the consultation (11 July 2011 and 5 September 2011) to consider these reports, recommendations and suggested redrafting.

2.2.6 Plans for the Finalisation of the Guidance

In August 2011, the GMC was the process of redrafting the guidance to take account of the Working Group’s decisions at the 11 July 2011 meeting. The intention is for the revised guidance to be approved by the GMC in late 2011 and to publish the new guidance in January 2012.



2.3 BACKGROUND TO THE AUDIT

The Social Research Centre was appointed in September 2011.

2.4 TERMS OF REFERENCE FOR THE AUDIT

The terms of reference of the audit specified that: The purpose of the audit was twofold:

- To assess compliance with the procedure for collating, analysing and

presenting information gathered as part of the formal consultation process; and,

- To consider whether the analysis reports the team produced reflected a fair and reasonable interpretation of the consultation data.

The scope of the audit was to evaluate the following:

Preparation of data for analysis (including the transfer of unstructured responses

into the structured questionnaire and the accuracy of replicating responses onto the database);

Process for analysis (instructions given and method for combining written

responses and discussion at meetings);

Interpretation of results (qualitative and quantitative data); and,

Recommendations arising from the interpretation of results. Consequently, the crucial test within this audit was the extent to which the data, findings, conclusions, recommendations which SRC generated concurred with those of the GMC Analysis Team in terms of completeness, accuracy and substance. NOTE: The reader should be aware that the brief issued by GMC for this exercise, made it clear that consideration of wider issues pertaining to consultation and involvement was not required as part of this assignment, “It does not fall within the terms of this audit to assess the broader GMC policy on consultation and engagement [this] will be considered as part of the GMC’s evaluation project.”

2.5 SCALE OF THE AUDIT

GMC commissioned SRC to invest five working days on the conduct of the audit of the consultation responses including this report on its findings.



3. WHAT WE DID 3.1 OVERVIEW

This Section details how each phase of the audit was conducted.

3.2 A PRINCIPLES-BASED APROACH

Our conduct of the audit was founded on the following six principles:

RISK Assessing where was the greatest risks lay in terms of any potential for the data to be altered or misinterpreted in any way, and then and focusing attention specifically on the higher risk areas.

SALIENCE Allowing the shape and emphasis within our audit to be informed by an understanding of the aspects of the consultation which were perceived to have special strategic significance for GMC.

ACCURACY Designing and deploying specific measures to check the extent to which the response has been correctly assigned to the appropriate section of the consultation in accordance with the analysis instructions.

COMPLETENESS Designing and deploying specific approaches to test the extent to which all information submitted had been included in the analysis.

SUBSTANCE Confirming, to the fullest possible extent, within the time and budgetary constraints of this exercise, that all material issues had been identified and considered.

SAMPLING Judicious sampling of key data sets to perform the above tests and checks. Note, within this audit, SRC sampled data both ‘vertically’ i.e. looking at how complete and accurate individual responses were (Data Sets 1 and 2 below) and ‘horizontally’ i.e. examining the interpretation of a specific segment of data in relation to specific questions.



3.3 INITIAL RISK ASSESSMENT & SPECIFIC AUDIT MEASURES DEPLOYED

At our initial meeting with the GMC Analysis Team on 22nd September 2011, the following data sets were identified and the risks associated with each were agreed. As the description below explained, the assessment of risk guided where SRC invested its efforts across the audit exercise. ‘VERTICAL’ DATA SETS - Looking ‘down’ the data… Data Set 1: On Line Questionnaire Responses

These were responses that had been submitted online directly by respondent (A total of 90 responses were submitted this way). In terms of ‘risk’ (i.e. perceived degree of threat to accuracy and completeness of the data) the online responses were considered to be NO RISK because the data had been entered directly by the respondent into a secure system and no interference with the data entered was possible. Hence, we did not audit this data in terms of its being entered completely and accurately since it was assumed to be complete and accurate by virtue of the data capture mechanism.

Data Set 2: Hard Copy Questionnaire Responses That Were Entered Manually

In total, 22 (check) hard copy responses were received. These were entered manually. SRC and GMC concurred that given that the data was highly structured and unambiguous, the risk of the data being entered incompletely or inaccurately was LOW. Hence, we analysed two randomly selected examples of such entries, one response to the long questionnaire and one to the short questionnaire.

Data Set 3: ‘Open ended’ written submissions

In total, 17 hard written submissions (in free text format) were received by GMC. The GMC Analysis Team had marked the hardcopies to show which questions (in the long questionnaire), they considered they pertained. Given the relative complexity of this type of data, and the possibility that some data could have been missed or incorrectly to assign a risk of ‘MEDIUM’ to this. Consequently, SRC assessed the completeness and accuracy of this aspect of data preparation by: • Printing off every one of the open ended responses (n=17); • Checking, in each case, that the various points made had been:

o Logically assigned to the correct question on the appropriate questionnaire

(our test for ‘accuracy’);



o Physically added to the correct part of the database (our test for ‘completeness’). This was done by comparing the entries on the original open-ended response with the entries found in ‘Community People’.

Data Set 4: Meeting Notes It was understood, that the fundamental purpose of analysing the meeting notes was to complement the feedback from the questionnaires and to identify any further substantive points. As the GMC’s own Instructions for analysis of the meeting notes stated, “you are looking for any new or additional points that were not made by respondents to the written consultation or any points which particularly support any common view in the written consultation”. The meeting notes were a diverse and complex set of data and inherently posed a number of challenges for a variety of reasons: The level of control - The meetings attended were mostly meetings being hosted by

other organisations or Directorates. Consequently, the GMC had limited control over the agenda for the discussion or the time available for discussion. Typically, the discussion time was short (less than 20 mins);

The format - The free format of receiving feedback necessarily makes for highly

heterogeneous data sets which are subsequently more complex to analyse unless a highly standardised approach is taken (More on this later, See Section 5).

Individual vs group view - During meetings, operating under such tight constraints, it

can be immensely challenging to separate out the view of an individual from that of a ‘group’ view. We note this had been done in some of the meeting notes but not all. This again, potentially makes subsequent analysis more complex.

Notwithstanding all of the above, and within the time and budgetary constraints of this exercise, the key issue for the audit was to discern if additional substantive points had been raised and whether or not these had been included in the GMC Analysis Team’s write up of the meetings. Given the significant number of questionnaire responses analysed, relative to the number of meetings conducted, it was agreed that the meeting data would be assigned a risk of ‘LOW’. Hence, SRC randomly selected two of the meetings and we highlighted and summarised what we considered to be the substantive points raised within each. We then checked whether these points were represented in the full analysis compiled by the GMC Analysis Team.



‘HORIZONTAL’ DATA SETS - Looking ‘across’ the data…

Data Set 5: Audit of Individual Questions The complete and correct interpretation of the data pertaining to specific questions was at the heart of the consultation and, consequently, was the major focus of the audit process. Given the scale and complexity of the data that had been collected, and the limited time and budgetary constraints available for the audit exercise, GMC and SRC mutually agreed that a sampling approach was the only feasible way forward. Whilst one option would have been for SRC to simply choose two questions at random to analyse in details, we rejected this (GMC agreed) on the basis that such an approach might not select areas for examination that the GMC Analysis Team considered to be the most pertinent and / or valuable in the context of an audit. Hence, an important opportunity to provide assurance and potentially rigour would have been missed. Consequently, in order to ensure that SRC’s audit work would be invested in examining aspects that the GMC Analysis Team considered the most valuable, we invited the Team to specify two areas that they considered to be especially salient in the consultation overall. Whilst several areas were potential candidates, the two areas finally selected were: • Resources – Specifically, Question 8 in the long questionnaire and Questions 3c and

3d of the short questionnaire; and, • Raising Concerns – Specifically, Question 21 in the long questionnaire, which was

linked to Question 7a of the short questionnaire.

The way in which SRC analysed and interpreted the data in relation to each of these questions was as follows:

• Step 1: Analyse and Interpret - In this Step, we created an analysis framework using

the same format as that of the Team i.e. Statistical tables, what was found, what it means etc. We then produced our findings, conclusions and recommendations etc by going directly back to the data within the databases and, crucially, without reference to the GMC Analysis Team’s own workings.

• Step 2: Compare - When we had completed our own independent analysis and

interpretation, we then compared this with the Team’s originals. The findings from our comparison are set out in Section 4).

The reader should note that whilst it was feasible and appropriate to compare the quantitative data from a strictly arithmetic perspective, qualitative data, by its very nature requires a different approach. Within the time and budgetary constraints available for this audit, SRC’s approach to analysing the qualitative data was to compare on ‘substance’ and ‘reasonableness’. This involved:



o Identifying the distinctive points2 emerging from each of the segments of

analysis (i.e. those who said ‘Yes’; those who ‘said ‘No’ and those who were ‘Not Sure’, comments where the main response was blank etc);

o Comparing this with the GMC Analysis Team’s own work to ascertain if there

were any material differences between our findings and theirs.

The reader should note that SRC is entirely open about the fact that its capacity to interpret the data and make recommendations based on it is heavily constrained. We did not, nor were we expected to, possess the level of specialist and contextual knowledge of the GMC Analysis Team. Consequently, our interpretation of the material is based only on a broad comparison of the distinctive points emerging from the data available for analysis and not on wider contextual knowledge or understanding of the full authority of and constraints upon GMC, legislatively, financially or otherwise. Furthermore, we note that in the GMC Analysis Team’s own instructions for analysing the data its states that the GMC recognised that further data needed to be considered before recommendations could be made “we [GMC] are not looking for concrete recommendations as we have other data sources to analyse and the final recommendations must take account of all strands of the consultation. But if in the course of your analysis, you find you have received clear messages such as drafting or things that are missing from the guidance, please feel free to note these as bullet points in the template under ‘Preliminary recommendations’. Given both of these factors, SRC’s approach to auditing the ‘discussion’ and ‘recommendations’ section of the GMC Team’s Analysis was to create a summary of our own of all of the distinctive points that had emerged. We then compared this with the points made by the GMC Analysis Team in the ‘discussion’ and ‘recommendation’ sections of its respective reports.

2 Given the budgetary and time constraints on this exercise, SRC determined that this was the only feasible approach to take. More detailed analysis by ‘themes’ and ‘mentions’ and / or detailed coding and quantification of the qualitative responses was not achievable within the limited specified for this exercise and indeed any such comparison would have involved pre-agreement with GMC on what the specific themes were (i.e. so as coding frames could be set up)



4. WHAT WE FOUND 4.1 OVERVIEW

This Section itemises the key findings, conclusions and recommendations that ensued from SRC’s examination of the data.

Overall, the audit had found that the GMC Analysis Team’s approach to analysis and interpretation was both thorough and compliant with the instructions given.

There were very high levels of completeness and accuracy throughout the

data sets examined with virtually all substantive points included.

In so far as SRC can judge, the correct interpretation has been applied to all of the points raised.

We have listed a number of general points (See Section 4.3 and 4.3) which

GMC may wish to reflect upon for regarding the analysis of data in future consultation exercises.

Data Set Audit Result Hard Copy Questionnaire Responses That Were Entered Manually

Very high levels of completeness and accuracy.

‘Open ended’ written submissions

Very high levels of completeness and accuracy - Average of 98% on both aspects.

We would stress that the specific aspects that created an apparent shortfall short fall below 100% are not material.

M

eeting Notes tive points were identified in the sample examined All substan

Audit of Individual Questions

ific aspects of different are

Virtually all of substantive points were identified in the sample examined

e would stress that the number of specWminimal and were not material. (See Section 4.2. and 4.3 below)



4.2

m Q8 – Analysis of the Comments Pos

stinctive points identified by SRC had been identified by GMC’s

Poi

o . SRC also

analysed the comments from respondents where the answer to the corresponding que o potent ctive points were raised, these were:

The need for engagement of local structures. It was perceived that these

to

sentative of the

Analysis Team that the first of these was identified within many other parts of the consultation response and has been included in the considerations overall. At the time

omments

os

those of SRC.

Virtually of the distinctive points identified by SRC had been identified by GMC’s Analysis Team. (See below)

ANALYSIS OF THE COMMENTS ON ‘RESOURCES’

Findings fro

itive Findings

Virtually all of the diAnalysis Team.

nts for Reflection

The GMC Analysis Team focused on analysing comments from respondents whhad answered ‘yes’, ‘no’ or ‘not sure’ to corresponding question

stion was blank. Whilst only two additional comments were identified, twially3 additional distin

o Advised against separating all doctors from doctors with additional responsibilities; and,

ounderdeveloped in NI and the question was raised was it therefore unfairimpose?

However, we understand from speaking subsequently with a repre

of writing, the Analysis Team was investigating the latter of these.

Findings from Q3c and 3d – Analysis of Statistics and CP

itive Findings

Perfect arithmetic match between the figures presented in all of the tables computed by the Team and

3 We say ‘potentially’ because we could not confirm whether or not these points were raised elsewhere in the consultation responses overall.



Points for Reflection

SRC was unclear why the statistics from ShortQ3c were not analysed by source in

the way other questions had been.

There were, potentially4 three additional distinctive points identified by SRC which were not mentioned by GMC’s Analysis Team, namely:

o Need for political honesty regarding resource allocation; o A suggestion to establish a new profession of health care managers who

would accept an ethical code; and, o A suggestion that patients should be given an address to write to / contact

their MP. Again, we understand from speaking with a representative of the Analysis Team that these points were not appropriate to include within the feedback from the consultation because they were beyond the remit of the GMC. Furthermore, such points did not occur frequently in the feedback.

Discussion and Recommendations from Long Q8 and Short Q3c and 3d

Positive Findings

Based on an examination of the data alone (and no other considerations), SRC is of the view that all of the substantive points from the quantitative and qualitative analysis have been identified and considered appropriately in the ‘discussion’ section of the GMC Analysis Team’s report.

We consider that the recommendations made are meaningful in the context of thdata examined and would seem to SRC to represent a reasonable way forward.

e

4 We say ‘potentially’ because we could not confirm whether or not these points were raised elsewhere in the consultation responses overall.



Findings from Q21 – Analysis of the Comments Positive Findings

All of the distinctive points identified by SRC had been identified by GMC’s Analysis Team.


None identified.

Findings from Q7a - Analysis of Statistics and Comments

Positive Findings

Perfect arithmetic match between the figures presentcomputed by the GMC Analysis Team and those of SRC.

ed in all of the tables

e distinctive points identified by SRC had been identified by GMC’s Analysis Team.

Points for Reflection?

s from Short 7a were not analysed by source in the way other questions had been.

mendations from Long Q8 and Short Q3c and 3d

ositive Findings

red appropriately in the ‘discussion’

We consider that the recommendations made are meaningful in the context of the data examined and would seem to SRC to represent a reasonable way forward.

All of th

SRC was unclear why the statistic

Discussion and Recom

P

Based on an examination of the data alone (and no other considerations), SRC is of the view that all of the substantive points from the quantitative and qualitative analysis have been identified and considesection of the GMC Analysis Team’s report.



4.3 GENERAL POINTS ON ALL STATISTICAL TABLES EXAMINED

Positive Findings

Perfect arithmetic match between the figures presented in all of the tables computed by the Team and those of SRC. The one point of difference we did identify was concerned with minor differences in the classifications of small segments of the data set. However, we would stress that this was not a material issue.


Notwithstanding this, the analysis of the statistics tables highlighted a number of points which GMC may wish to consider in terms of further refining its approach for future consultations. These are: Consideration of non-responses – SRC’s own preference is to cite the data for all

aspects of a response, including non-responses. This provides the reader with a fuller ‘picture’ of the data set and provides opportunities for learning (e.g. if a large proportion of responses to a specific question are blank).

Consideration of data that is not ‘categorised’ – Again, SRC’s own preference, as a

matter of practice, is to be explicit about this and we have done so on our own working papers showing our analysis of the statistics. Whilst in the case of this consultation this factor turns out not to be material (since only 6 responses out of 115 were not able to be assigned to a specific organisation or individual / role and of these 4 were ‘blank’) it is nevertheless a best practice approach.

Use of percentages – Whilst the percentages calculated by the GMC Team are all arithmetically correct, SRC’s own preference would be to not to compute or display such percentages at all when the actual numbers of responses in a particular category are small5.

Consistency of categorisation – This is a very minor point. In almost all cases, the

coding of the data on the statistics tables correlated directly on a one to one basis with what was recorded on the databases. The only exception appeared to be the category of ‘Other healthcare professionals’. As it turned out, the data shown on the Team’s table appeared to be a combination of two data categories on the database, namely, 'Other health care professionals' and 'Health Care Professionals or staff' (other than doctor). Again, for transparency and clarity – both internally within GMC and, where appropriate, externally - our preference would have been for single coding structure for this data, rather than a combination of two. We understand that GMC has now done this.

5 A widely used convention is that percentages are not shown where the number of responses is 30 or fewer.



4.4 TS ON QUALITATIVE DATA EXAMINED

s inherently a subjective process, and consequently, SRC fully expected that a level of apparent difference would arise

quence of this.

o it

futu

•

e up to replace general wording such as ‘a small number’; ‘several’ etc with the actual number of responses that applied in each and every case).

GENERAL POIN Positive Findings

With very few exceptions, the points identified by the GMC Analysis Team were substantively the same points as those identified independently by SRC. NOTE: We stress that any choice of themes i

as a direct conse


N tw hstanding the above, we offer the following points for GMC’s consideration in re consultation exercises:

Coding frames to analyse qualitative data - As indicated in Section 3 of our methodology, there was no opportunity, within the budgetary and time constraints set for this exercise, to carry out a fully ‘coded’6 and detailed analysis of the comments made. However, had such analysis been possible, it would have enabled GMC to a) separate ‘themes’ (i.e. recurring citing of specific points) from ‘mentions’ (e.g. cited one respondent only) and b) elucidate nuances and ranges of opinion within a theme. Whilst SRC considers that the substantive points have been elucidated and articulated using the processes applied by GMC’s Analysis Team, this practice may be one to consider in future exercises since it would enable the write-up on the analysis of the qualitative comments to be ‘ranked’ so that the most frequently cited issues are listed at the start, and the less frequently cited issues listed at the end. This approach is not without limitations since a slavish reliance on ‘frequencies’ to denote importance is not always appropriate. For example, it may be that the salience of the content (however much or little a particular point is cited) matters more, and hence, ‘salience’, rather than frequency of citing, becomes the characteristic that drives the ranking. We mention these options because we are conscious that, whilst the write up of the findings appears to SRC to be substantively correct and complete, we are aware that a fully coded analysis of the comments could have provided the basis for full traceability in terms of the analysis performed. (However, if such traceability already exists, coding would add further clarity and rigour to the writ

6 i.e. Whereby individual comments – and / or parts of comments - are assigned to specific themes so that the number and nature of the comments assigned to each theme can be identified and analysed in further detail.

Documents

Audit of Consultation Good Management Practice: guidance for All