The replicability problem

Although replicating successful health care interventions in new contexts is clearly essential for maximising the benefits of new ideas and innovations for patients, it is also a well-recognised challenge. A commonly seen phenomenon is that when initially successful interventions are spread to new settings they may fail to achieve the same impact, or indeed any impact at all.

The phenomenon of surgical safety checklists provides an example of these variable fortunes. While the introduction of such checklists around the world has sometimes been associated with significant reductions in surgical complications and mortality, in other cases it has not led to improvements, even when compliance with the checklist has been high.

On other occasions, even when some impact is achieved, there can be a reduction in effectiveness or ‘voltage drop’ as initiatives are replicated. In the field of social programmes more broadly, this phenomenon was nicknamed the ‘Iron Law’ by evaluator Peter Rossi in the 1980s, who argued that as a new initiative is implemented across more and more settings, the impact will tend toward zero.

There may sometimes be straightforward explanations for this ‘replicability problem’, such as a failure by adopters to adhere to the intervention protocols. In recent years, however, there has been growing interest in a deeper set of explanations: that we may not be describing interventions in ways that enable them to be successfully reproduced in new contexts, and that we may not be designing programmes to spread interventions in ways that adequately support adopters to reproduce them.

This problem of replicability is different from the one that has traditionally preoccupied innovation research, namely, identifying what drives the uptake of new ideas. Much of this wider research stems from Everett Rogers’ seminal work on the diffusion of innovations, first published in 1962, which explored the properties of innovations and social systems affecting uptake. A more recent and highly influential contribution is the review of the literature on the diffusion of innovations in service organisations by Trish Greenhalgh and colleagues, who highlight the roles played by a large range of factors in driving the adoption of innovation in health services. Here, by contrast, we are concerned not simply with uptake but with the challenge of successful or effective uptake and the factors that affect whether someone can replicate an intervention’s impact when they do take it up.

Perhaps it is not surprising that this proves such a challenge. For an adopter to be able to reproduce an intervention successfully, they need to understand how they can translate the idea into their own setting, they need to know just what matters for the intervention to work in this new setting, and they need to have the opportunity, motivation and capability to implement it. None of these things are trivial, yet it is surprising how often they are taken for granted in initiatives to scale up and spread new ideas.

This goes beyond the well-known phenomenon of incomplete descriptions of interventions, with crucial details often omitted from reports, hampering effective replication. For example, Hoffman and colleagues analysed reports from a large sample of randomised trials of non-drug interventions and found that more than half (61%) were not described in sufficient detail to enable replication of the intervention in practice. Rather, the issue may be whether we are conceptualising the interventions themselves in the right way. For example, is the conceptualisation broad enough to incorporate all those aspects of context that might impinge upon the intervention’s effectiveness? Or does it strike the right balance between the ‘what’ and the ‘how’?

And these challenges become especially acute with complex health care interventions, particularly innovations in clinical processes and pathways of the kind that the Health Foundation has supported over many years, such as improvements in patient flow or hospital discharge. This includes the use of new technologies such as apps and medical devices; these are sometimes viewed as ‘simple’ interventions, but their effectiveness will depend on the skills, behaviours and cultures of those using them, and the evolution of new ways of working to maximise their benefits.

This report presents lessons and insights on how to tackle these challenges that have emerged from Health Foundation programmes and research. It begins by considering why complexity poses problems for conceptualising and replicating interventions, before moving on to consider the implications both for how we describe interventions and how we support their spread.

The Health Foundation’s programmes to support scaling and spread

Over the last decade, the Health Foundation has run five major programmes to support the spread of innovations and improvements in health care. These have included programmes explicitly designed to support teams to spread complex interventions to new sites (such as the Scaling Up programme), as well as programmes seeking to improve health care at scale more generally (such as the Closing the Gap series of programmes) but which nevertheless included projects of the kind we are concerned with here (namely, those engaged in spreading defined interventions to specific sites).

Table 1 gives the details of these programmes dating from 2009 to 2017. Since then, a third round of the Scaling Up initiative has been launched, supporting a further seven teams.

Table 1: Details of Health Foundation programmes on scaling and spread 2009–2017

Programme

When it ran

Aim

Investment (including evaluation)

Number of projects supported

Closing the Gap through Clinical Communities

2009–2012

To support the uptake of improvement interventions through established clinical networks

£5.3m

11

Closing the Gap through Changing Relationships

2010–2013

To support organisations to implement interventions that improve the relationship between people and health services

£3.9m

7

Closing the Gap in Patient Safety

2014–2016

To scale evidence-based patient safety interventions through groups of collaborating organisations

£4.0m

9

Spreading Improvement

2014–2017

To support teams to spread their improvement interventions through creating contexts and infrastructures that support diffusion

£2.2m

5

Scaling Up (Rounds 1 & 2)

2014–2017

To support teams to implement tested improvement interventions at scale, in partnership with organisations that can support regional or national spread

£7.4m

13

With one team receiving a grant in more than one of these programmes, there are a total of 44 discrete projects across the five programmes, listed in Appendix 1.

How we have learned from the Health Foundation’s programmes

The insights presented in this report emerged from interviews with many Health Foundation grant holders and partners during 2017 and 2018.

We also benefited from close engagement with several of the projects supported by the Health Foundation’s programmes, through interviews with innovators, adopters and evaluators conducted in 2017 and 2018. Three of these projects are presented as detailed case studies in this report:

  • Shared Haemodialysis Care, a project to spread an approach to haemodialysis that gives patients the opportunity to take a greater role in their own care, beginning in 12 dialysis units and subsequently spreading to a further seven units in England between 2016 and 2018, as part of the Health Foundation’s Scaling Up programme (see Chapter 3).
  • Respiratory Innovation: Promoting Positive Life Experience (RIPPLE), a project to spread a new model of community clinic for tackling social isolation and improving health for people with chronic obstructive pulmonary disease (COPD) to six sites across the East and West Midlands between 2015 and 2018, as part of the Health Foundation’s Spreading Improvement programme (see Chapter 4).
  • Situational Awareness For Everyone (SAFE), a project to implement patient safety huddles – initially in 12 paediatric units and subsequently in a further 16 units in England between 2014 and 2017 – as part of the Health Foundation’s Closing the Gap in Patient Safety programme (see Chapter 5).

In addition, we conducted a survey of innovators and adopters from the Health Foundation’s spread-related programmes during December 2017 and January 2018, the results of which are presented at various points throughout this report. Further information on the survey is given in Box 1.

Finally, throughout the report, we draw on examples and evaluations from the Health Foundation’s improvement work more broadly, both from the spread-related programmes listed above and other relevant programmes such as Safer Clinical Systems (2008–2016) and Flow Cost Quality (2010–2012).

Box 1: Our survey of innovators and adopters from the Health Foundation’s programmes

Over the last decade, the Health Foundation has funded five major spread-related programmes, which together have supported 44 different projects. After analysing each of these projects, we concluded that 26 could be categorised as aiming to spread a defined intervention or approach to specific adopter sites – the paradigm of scaling and spread with which this report is concerned. The other 18 projects took different approaches to achieving improvements at scale, such as by influencing national guidelines or through improvement collaboratives supporting different interventions in different sites.

To investigate the views of programme participants, we conducted surveys of the innovators and, separately, the adopters from these 26 projects, throughout December 2017 and January 2018, using Qualtrics software. We received responses from 21 innovators and 42 adopters.

The survey asked them about their experiences of trying to spread or implement an intervention as part of the relevant Health Foundation programme; sought their reflections on the process with hindsight; and invited their views on general questions about spread and adoption within the context of their experiences in the Health Foundation programme. The results are illustrated at various points throughout this report.

Previous Next