The role of improvement during the

response to COVID-19: Emergent

findings from Q’s Insight Survey 2020 Working paper Matthew Hill, Henry Cann and Jo Scott November 2020

This working paper outlines emergent findings from Q’s annual insight survey and supports the event session: What Q said about improvement during COVID-19: highlights from Q’s annual insight survey. Our analysis will continue to develop including as we undertake in-depth follow up interviews. We are sharing the emergent findings with you now to test our analysis and hear what resonates or what you would challenge. Please do get in touch to share your thoughts.

2

Summary of key lessons for improvement Q’s annual insight survey asked members what the role of improvement has been in supporting change during COVID-19 and what this means for the future of improvement. We received 225 responses between August and September. The responses have allowed us to draw some clear conclusions, especially as some respondents gave considerable detail in free text responses. However, the relatively low number of responses overall has limited any breakdown analysis and means some of the conclusions are more tentative. This summary box presents 13 key lessons for improvers and those supporting improvement. The rest of the report gives more detail to support these key lessons.

1. Improvement tools, methods, approaches and mindsets played an important role for a majority of respondents and this role was generally seen to have increased in their work, their team’s work and their organisation’s work.

2. Improvement was used extensively for rapidly reviewing and improving

processes and practice with the Model for Improvement– specifically Plan Do Study Act (PDSA) cycles – considered particularly useful due to the pace of change and need for innovation.

3. Another widespread application of improvement was for engaging staff,

driven by the need to systematically involve a wide range of stakeholders, ensuring buy-in and enabling group reflection. Popular tools that supported open and frequent communication included Big Rooms, huddles and the use of Liberating Structures.

4. Considerable challenges were seen in applying measurement for improvement. Respondents faced difficulties around reduced data collection, high levels of uncertainty when isolating variation, and the sheer speed of change meaning that evidence was not always integrated into decision-making processes.

5. Respondents reported relatively little use of improvement to engage patients. This was due to lack of time, lockdown restrictions and the perceived inadequacy of remote forms of patient engagement.

6. Improvement appears to have played a more important, valuable and strategic role during COVID-19 in organisations that had a well-developed approach to improvement pre-pandemic.

7. The enablers of improvement included supportive and empowering leadership, staff capacity and willingness to get involved with improvement, as well as cultural shifts such as a reduction in bureaucratic constraints. A key barrier was ‘command and control’ style leadership, which had intensified for some during COVID-19.

8. Overall the responses suggest that much improvement took a distinct form during the pandemic response, organised around short-term goals and planning. It was often used with flexibility and sometimes applied in a limited or partial way. Some found these approaches helpful to achieve improvement goals at pace, whereas others questioned if this may undermine the longer-term sustainability of the changes made.

3

Introduction and focus of the survey COVID-19 has precipitated unprecedented changes in service delivery and ways of working across health and care. Right from the start of the pandemic, competing narratives emerged about the role of improvement in these changes. Some experiences suggested it was improvement’s moment to come to the fore whereas others suggested that certain core elements of improvement were being bypassedi,ii,iii. To understand the role of improvement in more detail - and to ensure that individual improvers and we, as an improvement community, learn the right lessons for the future - we undertook a survey of Q members’ experiences of the COVID-19 response. This working paper presents the key emergent findings from the survey. Some of the findings are clear and, we believe, have clear implications for improvement practice

9. We borrowed the concept of a ‘crisis standard of care’ and asked whether the

responses can be viewed through the lens of a ‘crisis standard of improvement’. Here, the constraints of the crisis context are integrated into our judgements of ‘good’ improvement practice during the pandemic. Through this lens it is clear that some respondents applied a more effective and sophisticated ‘crisis standard of improvement’ than others.

10. For some, more technical, rigid and niche improvement methods were seen as unhelpful, with the dominant view amongst respondents being that broader improvement mindsets and principles were more important in underpinning positive change. In interpreting the findings, we also argue that it is useful to frame improvement as functioning like a ‘muscle’ during the pandemic response, where those with a bigger and more supple improvement muscle were able to apply it more fully and flexibly.

11. Improvers were able to access support from a range of sources during this period. Most frequently, they accessed support from within their own organisations. Rather than more technical written resources, they tended to value support accessed directly from others, including informal conversations with others facing similar challenges, and using online chat groups or structured reflective sessions with others.

12. Respondents identified the top three issues that the improvement community should prioritise as ‘working in a more integrated way across teams and organisations’, ‘introducing digital innovations effectively’ and ‘embedding systematic approaches to improvement within health and care organisations’.

13. Finally, the survey asked what the biggest challenges for further embedding improvement in health and care’s response to COVID-19 were. Responses included the importance of learning the right lessons from the pandemic, continuing to support collaboration, promoting enabling leadership and devoting sufficient resources to improvement. Respondents also identified specific challenges for improvement including ensuring it captures the potential of digital by adapting tools and building capabilities. Respondents also made calls on leaders to ensure that improvement more directly contributes to system priorities as well as work on the frontline.

4

and policy. We also present some more tentative findings, which we believe enhance our understanding of the role of improvement. Partly due to these more tentative conclusions and the relatively low number of responses we received, the findings in this report should be positioned as an important contribution to the conversation on improvement’s role, rather than the final word. Indeed, as we continue to refine our analysis and have some follow up interviews planned, we actively welcome your feedback so please do get in touch via email or on Twitter (@MatthewHillsays).

The survey method The online survey (using Qualtrics) was emailed to all Q members (3,880) between August and September 2020. It included a combination of 29 closed and free text questions, receiving 225 analysable responses. Respondents were broadly representative of Q members overall in terms of primary role type, organisation type and country of the UK. Quantitative analysis was undertaken using R and qualitative analysis was undertaken through developing inductive codes and applying thematic analysis directly in Excel and Word. Limitations include the relatively low number of responses, which has limited analysis that breaks down the respondents by different groups and the possibility of a non-response bias with more engaged members likely to be over-represented. So as to capture a wide range of experience some questions asked about ‘improvement tools, methods, approaches and mindsets’ and this broad framing has raised ambiguities and imprecision in interpreting some responsesiv.

The role of improvement Half (51%) of respondents felt that improvement had been very important in health and care generally during COVID-19, with 82% feeling it had been moderately or very important. Respondents in non-clinical roles were substantially more likely to say that improvement had been very important during COVID-19.

Overall more respondents said that the role of improvement had increased during the response to COVID-19 than said it had decreased or stayed the same. As figure 1 shows, this was true at the individual level (44%), team level (45%), organisational level (49%) and in health and care generally (46%).The survey explored some other dimensions of the role of improvement.

Figure 1: Did the role of improvement tools, methods, approaches and mindsets increase or decrease during the response to COVID-19?

5

As can be seen in figure 2, the vast majority of respondents agreed that they had been able to make good use of their improvement capabilities, that their organisation’s leadership had supported the role of improvement and that their team had applied a systematic improvement approach during the response to COVID-19. The survey also showed that the majority of respondents agreed that their team had

achieved a lot of positive change during COVID-19. Many factors will have

underpinned this positive change but interestingly those who assigned the highest

importance to the role of improvement were more likely to report achieving a lot of

positive change in this period.

How improvement was used and what was most challenging As can be seen from figure 3, a wide range of improvement tools, methods,

approaches and mindsets have been used for a broad variety of purposes during the

response to the pandemic. Much more detail on the value and challenges of different

elements of improvement was provided in the free text responses, with the key

findings drawn out in this section.

Figure 2: Agreement with statements regarding the role of improvement during COVID-19

6

Improvement was used extensively for rapidly reviewing and improving processes

and practice (70% to a great or moderate extent). The Model for Improvement,

specifically PDSA cycles, were the most cited tools for this. These were seen as

especially useful due to the increased need to reflect on what was and wasn’t

working well – especially at the start of the crisis. They were also valued for their

suitability for the context of rapid change and innovation and their simplicity. Others

mentioned the flexibility of PDSA cycles with one respondent finding them ‘easy to

adapt and use during COVID’. However, others discussed how PDSAs were applied

incompletely with some stages missing. One respondent noted that ‘people are

moving too fast to think about the learning generated before making their next plans’.

Other reflective learning tools used included After Action Reviews and agile

methodology.

Respondents also used improvement to a large extent for engaging staff during

COVID-19 (72% to a great or moderate extent). Responses stressed the importance

of this application to support change at speed and engage those who were required

to deliver on the change as quickly and substantially as possible. Tools that

supported staff resilience and psychological safety were seen as particularly useful

by some. A wide range of tools were cited including Big Room methodology for

engaging a wide range of stakeholders; Liberating Structures – including when

adapted for virtual meetings; and huddles.

Figure 3: The extent to which improvement tools, methods, approaches and mindsets were used for different purposes

7

A majority of respondents (60% to a great or moderate extent) used improvement for

measuring what’s happening. Often this involved relatively informal data capture with

staff although more systematic measurement was seen as especially useful where

there were relatively few key measures being used. However, the challenges of

effective measurement and evaluation were many. Firstly, for some, the systematic

measurement of variation had been difficult due to the unpredictability and lack of

control over variables in a chaotic crisis context. Secondly, for others, the issue was

that their organisation did not collect the right data, either because it was focused on

compliance rather than improvement or because, in response to the pandemic,

expectations around data capture had been reduced. Thirdly, respondents faced

challenges collecting data due to COVID restrictions (e.g. observational data).

Fourthly, some faced limited staff time to either collect or analyse evaluative data.

Fifthly, for some, the challenge was that priority was given to service delivery over

measurement with one respondent stating, ‘the focus is just on getting this 'done'

rather than systematically planning change and measuring the difference.’ Finally,

and perhaps most fundamentally, the sheer pace of change meant that

measurement of effectiveness or longer-term outcomes was not able to inform rapid

decision-making.

Improvement was less frequently used for engaging patients and carers – with less

than two-fifths (38%) using it for this to a great or moderate extent. Some positive

examples of remote engagement were offered. Yet it was clear that this element of

improvement was often missing due to the required speed of change and the

dependence on tools that had been designed for face-to-face engagement. One

respondent reflected that this meant they had ‘tended to fall back on traditional

(employed) expert patients as they have been easier to access and engage’.

Across the different applications of improvement there were competing perspectives

around the potential for digital and remote adaptations of improvement tools. Some

respondents reflected that remote working was a barrier to improvement, and felt

that it is second best to face-to-face engagement. Others felt that it could be as good

as, or better than, face-to-face in the longer term as it removes geographical

boundaries and allows for more flexible engagement. Indeed, some respondents had

successfully adapted tools such as Liberating Structures and huddles to work well

remotely during COVID. For improvement to capture this potential it will require

further adaptation of improvement tools, better platforms and more staff training.

Overall many respondents described how improvement took on a distinct form during

the COVID response as it was characterised and organised around short-term goals

and planning. It was also often used with flexibility and sometimes applied in a

limited or partial way. Some found these approaches helpful to achieve improvement

goals at pace, whereas others questioned if this may undermine the longer-term

sustainability of the changes made. More work is needed to explore the implications

of this, and the extent to which this was true for different phases of the pandemic

response.

8

Enablers and barriers of improvement

As can be seen in figure 4, leadership emerged as an important factor, either by

providing autonomy to teams (72% to a great or moderate extent), or by setting an

inclusive and compassionate approach (68% to a great or moderate extent). Some

respondents argued that an overall mindset change was driven by strong leadership

that empowered those on the frontline. This resonated well with work by Health

Foundation colleagues that characterised ‘top-down clarity and bottom-up

agency’v(p7) as a key enabler of positive change during COVID. As one respondent

described, there was ‘permission to try new things rapidly without having perfected

the approach’. However, in contrast, those who had faced greater challenge in

drawing on improvement often described a ‘command and control’ style leadership in

their organisation. As well as stifling a culture of testing and innovation, this was

seen to inhibit improvement tools that aim to engage stakeholders in diagnosing

issues and co-developing potential solutions. So, despite the survey results

supporting the wider positive narrative around enabling leadership during COVID-19,

the results also make clear that there were many examples where this was not the

case. More constraining forms of leadership persisted in some settings and, in some

cases, were intensified.

Figure 4: Enabling factors for improvement work during the response to COVID-19

9

Sixty-nine per cent of respondents reported their teams’ improvement efforts were

enabled to a great or moderate extent by reduced bureaucratic constraints during the

response to COVID-19. This was often wrapped up with a number of concurrent

shifts, including mindsets and processes. Also described were clearer and quicker

decision making, and fewer financial and procurement hurdles.

Although clarity of purpose is not identified as a top enabler in figure 4, respondents

described how the clarity of focus and shared priority provided by the urgency of the

crisis response helped to challenge assumptions and shift behaviour that was

conducive to improvement approaches. As one put it starkly ‘because we were all up

against it, all of the usual crap disappeared (inter-organisation power plays and bad

behaviour) and this was great.’

Staff capacity and willingness to engage (72% to a great or moderate extent) was

another key enabler. Staff being ‘brave’ in trying things and not fearing failure, came

up frequently in relation to this.

As well as these broader enabling conditions, improvement appears to have played a more important, valuable and strategic role for those respondents who said their organisation had a well-developed approach to improvement pre-pandemic. Firstly, these respondents were more likely to have used improvement tools for planning for the future and deciding where to focus efforts. Secondly, they were more likely to report using it for engaging staff and enabling their teams to work effectively together. Thirdly, and finally, they were also more likely to agree that they had been able to make use of their personal capabilities, and that their team had applied a systematic approach. By considering these findings alongside the other bureaucratic and system enablers mentioned above, this highlights how important it is to continue to invest in and develop skills for improvement, while giving enough attention to influencing and enabling the system conditions in which improvement can thrive.

Was an effective ‘crisis standard of improvement’ applied? To help us interpret the findings overall, we borrowed from the concept of a ‘crisis

standard of care’. We asked whether responses suggested an effective ‘crisis

standard’ of improvement had been applied during the COVID response. By

integrating the crisis constraints - such as the speed of change, the high level of

uncertainty and remote working - into our assessments of what ‘good’ improvement

during a crisis is, it allows us to make more sophisticated assessments of the role of

improvement. In turn, this helps us learn the right lessons for the future. For

example, it is clear that given the crisis context, some approaches to measurement

for improvement – such as capturing long-term outcomes – were simply not possible.

However, it is also clear from the responses that some applied a much more

effective ‘crisis standard’ of measurement than others. So, although for some,

measurement was limited to relatively informal reflection with stakeholders, others

were still able to systematically capture data around engagement levels and

measure certain quality metrics such as staff and patient satisfaction. Similarly, as

outlined above, patient engagement by respondents was relatively low during COVID

and some clearly felt that meaningful participation was just not possible. Yet, others

10

proactively engaged patients and adapted their tools in order to ensure a higher

crisis standard of patient and public involvement in improvement. Finally, the

responses leave an open question as to what the most effective crisis standard of

the Model for Improvement is. As stated earlier, some applied this very flexibly and

successfully to their COVID-response work, whereas others felt that if its application

becomes too partial and limited then its value is lost.

It’s important to note that no respondents described explicitly applying ‘a crisis

standard’. However, we believe that a clear lesson from the responses is that a more

intentional approach to adapting improvement to meet the crisis constraints of

COVID led to a more advanced role for improvement. Indeed, there are clearly

lessons from the experience of COVID-19, which could be used to ensure a higher

standard of improvement is applied in future crises. These lessons are both from

specific examples of good crisis practice as well as the benefits of intentionally

applying a crisis standard more generally. We also think that by locating the

experiences of improvement squarely within the crisis context it may guard against

drawing overly simple conclusions for more normal times. For example, although the

partial and limited application of different forms of improvement may have

represented good practice given the constraints resulting from COVID, this does not

necessarily mean this is something to aspire to outside of the crisis response.

Were improvement methods, mindsets or muscles more important? Another more tentative finding on which we welcome feedback is the importance of formal improvement tools and methods, relative to broader improvement approaches and mindsets. As has been discussed, some respondents drew heavily and successfully on specific improvement tools. However, the dominant view amongst respondents was that it was broader improvement mindsets and principles that were most important in supporting positive change. Indeed, some felt that more technical, rigid and niche improvement methods were not useful at all during COVID. In attempting to understand the findings around the methods-mindset tension, we argue it is also useful to frame improvement as functioning like a ‘muscle’ during the pandemic response. This is primarily our interpretation of the results, but this type of framing did appear in some responses. For example, one respondent said, ‘What was encouraging was that you could observe methodologies being used, almost as a heuristic, rather than through any planned desire to follow a strict methodology.’ Another felt that ‘there simply wasn't time to devote to ensuring accurate measurement or implementation or even plan - but it was as if muscle memory kicked in for a lot of the team.’

Again, we think this framing is useful for helping us learn the right lessons from the

pandemic. Firstly, we heard that those with the most well-developed approach pre-

COVID – or you could say those with the biggest ‘improvement muscle’ – saw a

more important role for improvement during the crisis. It also appears from some of

the free text responses that those with what could be characterised as a more

‘supple’ improvement muscle were more able to flex their approach. Secondly, this

framing may overcome the method-mindset tension to some extent. Even if we

11

conclude that it was indeed mindsets that were most important during the crisis, part

of the value of learning and applying certain improvement tools and methods in

practice is that this helps build our improvement muscle. Once built, it can then be

flexed more easily to respond to different contexts.

Support for improvement Figure 5 outlines the sources of support that respondents accessed for their

improvement work. Our analysis found that during this time, respondents accessed

many different types of support from a range of sources. They were most likely to

have accessed useful support from their own organisations (59%), followed by

individuals outside of any formal network (40%). Thirty-one per cent of respondents

had accessed useful support informally through the Q community and twenty-six per

cent accessed this support from Q’s formal offers and infrastructure such as national

events, Q Visits and Special Interest Groups. Forty-four per cent of all respondents

had accessed support from one or more of these Q-related sources.

Figure 6: Types of improvement support that respondents and their teams found most useful during COVID

Figure 5: Types of improvement support accessed during COVID

12

Figure 6 shows the types of support for improvement work that respondents found

most valuable (respondents could choose up to three). The most valuable are those

based around direct interaction with others. Fifty-one per cent selected informal

opportunities to discuss with others who had experienced similar challenges, and

thirty-three per cent selected online chat groups or threads. Structured opportunities

for reflection were also considered most valuable by almost a third (30%). Somewhat

less valuable to respondents seems to have been more technical resources

including written resources on improvement tools, written examples of applied

improvement or training in improvement tools, methods and approaches.

Future priorities and challenges for improvement The final section of the survey looked to the future. Figure 7 shows results from a

question that asked Q members to select the wider system priorities that they think

the improvement community should focus on contributing to. The question presented

a list of key issues within health and care and asked respondents to prioritise up to

three. The top three were: working in a more integrated way across teams and

organisations (41%), introducing digital innovations effectively (38%) and embedding

systematic approaches to improvement within health and care organisations (38%).

These responses provide useful insights into member priorities. However, additional

work is needed to understand more fully what these issues really mean to the

improvement community and what members think the distinctive contribution of

improvement is to supporting these system priorities.

Figure 7: Issues that the improvement community should prioritise in the future

13

Respondents provided rich insights around the biggest challenges to embedding

improvement in health and care’s response to COVID over the next 12 months. They

also provided some suggested solutions for overcoming these. Some of the

challenges largely pre-date the pandemic and will be familiar, whereas others have

taken on a new colour and prominence. Challenges relating to both broader enablers

of improvement and challenges for improvement itself were identified.

Generally, there was a call to arms around ensuring the system really understands

what conditions have made rapid change possible during COVID. Indeed, there were

some fears that these conditions were already regressing, including: the

reintroduction of competitive models of operation at the expense of positive

collaboration; the return of managerial approaches at the expense of enabling

leadership; reduced staff time to engage in change as demand returns; and a drying

up of some of the financial resources that have been made available.

There were a range of challenges identified relating to workforce and morale. These

included staff feeling tired and tired of change, goodwill being ‘spent’, and staff

requiring support in new ways of working, especially digital. Part of the suggested

solution to these challenges was a renewed focus on staff wellbeing.

As well as stressing these broader enablers of improvement, respondents also

identified ways that improvement itself needs to change and adapt in order to meet

the challenges of the next twelve months (and beyond). There was a feeling for

some that improvement needs to be proactive to capture the potential of digital:

ensuing tools are fully adapted, that resources are devoted to technology and that

staff capabilities in remote work are built.

Further, as has been outlined, there is a feeling from some that more rigid, technical

and niche improvement tools and methods have not been useful during the

pandemic. In response, it was felt that language should be simplified and that there

is a need to consolidate around approaches that are more easily understood and

applied – especially where there are multiple tools that may appear distinct but are,

in fact, based on very similar fundamentals.

There were then a range of challenges identified around leadership for improvement

with a call from some for improvement to make a more direct contribution both to

system priorities and the priorities of those on the ‘frontline’. As one argued, ‘working

across the whole system, with shared methodologies and goals could pay dividends

for those systems.’

Finally, the most cited challenge, and one that draws together some of the earlier

findings about enablers and barriers, is the need for improvement to be embedded at

the core of health and care rather than operating on the fringes or being seen as

non-essential. Respondents outlined that this should be underpinned by: building

improvement capabilities across teams; offering protected staff time; having

improvement-friendly regulatory, governance and data systems; and having clear

expectations around the role of improvement from leaders. As one respondent said,

14

‘to build improvement capacity it needs to be integrated culturally and systematically.

Needs to be a national mindset that improvement work is not an optional extra

delivered by passionate staff in their own time but a core part of work.’

Emergent conclusions and next steps The survey responses show the important, and often increased, role that improvement has played across health and care’s response to COVID-19. It showed that improvement has been particularly useful for rapidly reviewing work and meaningfully engaging staff, with a range of tools and approaches applied. It has also shown that some core elements of improvement have been more limited in their application with particular challenges around systematic measurement and patient involvement. The survey identified several key enablers of effective improvement including the emergence of new forms of leadership, staff willingness to get involved and a range of useful improvement-specific support. Perhaps most interestingly, improvement appears to have played a more important, valuable and strategic role in organisations which had a well-developed approach to improvement pre-pandemic. To fully interpret the results, we applied two more tentative frames. Firstly, in an attempt to understand the flexible, more short-term and, sometimes limited nature of improvement during the pandemic, we explored to what extent an effective ‘crisis standard of improvement’ had been applied. Secondly, we argued that it is useful to frame improvement as functioning like a muscle during COVID-19: those with the biggest and most supple ‘improvement muscle’ were able to apply it in the most sophisticated and flexible way. Finally, the survey presented some of the future priorities and challenges for improvement. It highlighted the need to more directly contribute to system priorities and to ensure that the conditions that have enabled positive change during COVID-19 endure. It also revealed that those working in improvement must capitalise on the potential of digital and that some of the more technical, niche and rigid tools should be consolidated and simplified. Overall, it is critically important that individual improvers, and all of us in the improvement community, learn the right lessons from the pandemic and clearly and confidently communicate them to non-improvers across health and care. This working paper has presented the emergent lessons that we are drawing from our analysis, but we will continue to refine and develop these findings through further input from members and follow up interviews with a small number of survey respondents.

Please do get in touch by email or on Twitter (@MatthewHillsays) to share your thoughts on any aspect of this working paper – we would love to keep the conversation going with you and hear your views.

15

i Scott J, Hill M. Frontline insights on the rapid implementation of video consultations: what's needed now? Q, The Health Foundation. 2020 August. Available from: https://q.health.org.uk/news-story/video-consultations/ ii Lewis R, Pereira P, Thorlby R, Warburton W. Understanding and sustaining the health care service shifts accelerated by COVID-19. The Health Foundation. 2020 Sept. Available from: https://www.health.org.uk/publications/long-reads/understanding-and-sustaining-the-health-care-service-shifts-accelerated-by-COVID-19 iii Shah A, Pereira P, Tuma P. Quality improvement at times of crisis. (forthcoming) iv For ease of expression, the term ‘improvement’ is often used as shorthand for this broad definition throughout this paper although it is stated if a particular aspect of improvement is being referred to. v Lewis R, Pereira P, Thorlby R, Warburton W. Understanding and sustaining the health care service shifts accelerated by COVID-19. The Health Foundation. 2020 Sept. Available from: https://www.health.org.uk/publications/long-reads/understanding-and-sustaining-the-health-care-service-shifts-accelerated-by-COVID-19