The authors would like to thank a number of people who have provided helpful comments and insights during the production of this report. These include Amanda Begley, Ben Bray, Alan Davies, Jennifer Dixon, Mary Dixon Woods, Ruth Glassborow, Trish Greenhalgh, Nik Haliasos, Hugh Harvey, Kassandra Karpathakis, Josh Keith, Xiaoxuan Liu, Michael MacDonnell, Erik Mayer, Breid O’Brien, Sarah Jane Reed, Paul Sullivan, Richard Turnbull and Wenjuan Wang.

The views expressed in this report are those of the authors alone.

When referencing this publication please use the following URL: https://doi.org/10.37829/HF-2021-I03

1-1 What are automation and AI?

Automation is the use of technology to perform rule-based tasks with minimal human input. Encyclopaedia Britannica describes it as ‘performing a process by means of programmed commands combined with automatic feedback control to ensure proper execution of the instructions’, while the International Society of Automation defines it specifically as ‘the creation and application of technology to monitor and control the production and delivery of products and services’. While automation typically involves the execution of tasks previously done by humans, this is not necessarily a defining feature, with increasingly sophisticated technology enabling the automatic performance of tasks humans could not do, such as rapid analysis of large datasets.

Automation is a major theme in discourse about changing labour markets and the future of work. For many tasks, automated systems have the potential to improve on human performance, by reducing errors and improving productivity, and analysis conducted in 2019 by the Office for National Statistics (ONS) found that around 1.5 million jobs in England were at high risk of having some of their duties and tasks automated in future. This report will look at both the potential automation of tasks usually undertaken by health care workers, as well as the use of automation and AI to assist health care workers in performing tasks.

Advances in robotics and AI are extending the reach and capability of automation, both in the realm of physical tasks and increasingly the realm of cognitive tasks; as the AI Index puts it, ‘robotics puts computing into motion and gives machines autonomy. AI adds intelligence, giving machines the ability to reason’. A 2016 House of Commons Science and Technology Committee report describes AI as statistical tools and algorithms that ‘enable computers to simulate elements of human behaviour such as learning, reasoning and classification’. Recent years have seen advances in AI due to the increasing availability and quality of data, and improvements in technology and processing power. These include developments in machine learning, where algorithms are trained to make predictions using large datasets, and especially in ‘deep learning’, a type of machine learning using artificial neural networks. Among other things, these systems can learn to recognise and classify patterns in digital representations of sounds, images and text.

Given the significant overlap between these fields, this report will often refer to AI and robotics alongside automation, including situations where these technologies are used to automate some parts of a task (such as information analysis for decision making) but not others (such as decision selection).

1.1.1. Modes of automation: replacing versus assisting

It’s useful to distinguish between some different ways in which automation can relate to human task performance in health care.

Replacing: As highlighted above, in some cases automated systems are intended to perform tasks previously carried out by humans, replacing human input.

• This can happen where an automated system is able to perform tasks to a similar level to human workers (or at least to a ‘good enough’ standard), and so using an automated system to perform these tasks can free up health care staff to focus on other work. In Figure 1, this mode of automation is described as substituting for human input.
• On other occasions, the performance of an automated system might significantly exceed human capabilities, so by replacing human input it provides an opportunity to improve task performance (for example, where an automated system can execute tasks at much greater speed), and this may provide a rationale for automation independently of the benefits of releasing staff time. In Figure 1, this mode of automation is described as superseding human input.

Assisting: More commonly, automated systems can be used in health care to assist workers, rather than replace them.

• This can happen by using technology to automate just one component of a task or to provide additional capacity or functionality in a way that allows a worker to improve task performance – not because they can’t do what the technology is doing, but simply because having the technology effectively increases their capacity and allows them to focus on other aspects of task performance (for example, using dictation software to take notes). In Figure 1, this mode of automation is described as supporting human input.
• On other occasions, technologies designed to assist human task performance may also extend human capabilities, even though they are not intended to operate autonomously. For example, surgical robots may allow a greater degree of precision than humans alone, while AI-driven clinical decision support tools may exceed human information-processing capabilities. In Figure 1, this mode of automation is described as strengthening human input. Indeed, research suggests that while AI systems can match or even exceed humans in some ‘high end’ tasks (those requiring a high level of cognitive ability), when these systems are combined with human experience, intuition and knowledge, the impact of AI and robotics can be increased.

Figure 1 illustrates these different modes of automation. Note that the same technology could be used in different ways; the mode of automation will depend on how it is deployed on any particular occasion. It is worth noting that when technologies are deployed for supporting or substituting (the bottom row), the primary motivation is often to free up staff time, whereas when technologies are deployed for strengthening or superseding (the top row), the primary motivation is often to improve task performance.

Figure 1: Modes of automation

The use of automation to assist (and enhance) rather than replace is one way in which the impact of automation in health care can differ from other sectors. In industries such as agriculture and manufacturing, new technologies have often replaced labour (for example, the combine harvester or industrial robots for painting), a trend that continues today. A 2019 report by Oxford Economics estimates that around 1.7 million manufacturing jobs have been replaced by robots since 2000, including 400,000 in Europe. In health care, by contrast, new technologies have for the most part tended to supplement rather than replace labour, providing the means for health care workers to improve care or do their job more efficiently. This is partly because, as we explore in Chapter 3, many tasks in health care are difficult to automate. Instead, it is the potential of automation to assist NHS staff to manage high workloads that has attracted interest as demand for staff continues to increase.

1.1.2. Automation and the future labour market

While automation is not a new concept (see Box 3 for a brief history of automation in health care), the evolving capabilities of AI and robotics to undertake increasingly complex cognitive tasks, as well as an ever-growing range of manual tasks, have become a key consideration in analyses of how jobs and occupations could change in the coming decades.

Studies modelling the impact of automation on the future labour market have produced a wide range of estimates, but suggest the impact could be both far-reaching and unevenly distributed. For example, Frey and Osbourne estimate that 47% of occupations could be automated over the next 20 years, with roles in office and administrative support, production, transportation and service occupations highly susceptible. On the other hand, the Organisation for Economic Co-operation and Development (OECD) estimates that only 9% of occupations are at high risk of automation because many still contain a substantial share of tasks that are hard to automate. The ONS, investigating work in England, suggests that the impact of automation will vary across the labour market, with lower-skilled roles more susceptible to automation and women, young people and part-time workers most likely to work in roles that are at high risk of being automated.

Predicting the precise effects of automation on employment is difficult, however. The OECD study argues that even if a job is at high risk of automation, it will not necessarily result in job losses because workers can adapt by switching tasks and expanding their roles, and because technological change will also create new jobs. Similarly, a 2019 European Commission report argues it is unclear whether the net effect of automation will be job replacement or augmentation, given that AI and robotics will create new jobs as well as eliminate others.

Modelling also suggests the impact of automation will vary by sector. In line with the observations above that automation in health care has tended to supplement rather than replace labour, studies estimate the future impact of automation on jobs in health and care to be lower than in other sectors. For example, PricewaterhouseCoopers (PwC) estimates the proportion of jobs at high risk of automation in health care could rise from around 3% in the early 2020s to 20% by the mid-2030s, with financial services seeing the biggest effects over the short term and the transport sector over the longer term. The PwC model suggests the impact on jobs in health and care, as well as in education, will be lower than in other sectors.

In light of these trends, policymakers are grappling with how to respond to the labour market impacts of automation. The 2016 Taylor Review, for example, explored how employment practices need to change in order to keep pace with the modern economy. Specifically in relation to automation, the Review highlighted the importance of supporting people to gain appropriate skills for the future workplace – in particular, skills that are less likely to be affected by automation, such as relationship building, empathy and negotiation, which over time could become more valuable. It also highlighted the importance of lifelong learning to enable people to adapt to, and remain relevant in, a changing labour market.

1-2 Policy approaches to support automation and AI in health care

In health and care, policymakers are also focusing on helping providers and patients exploit the potential of new technologies. In England, the Department of Health and Social Care (DHSC) and NHS England and NHS Improvement have focused on several broad challenges relating to digital technology and information technology (IT). The aim in particular has been to ensure the NHS has the right basic infrastructure in place, with systems that are interoperable and enable the exchange of data, as well as addressing challenges around leadership and skills. Key initiatives include:

• The 2016 Wachter Review of health IT in secondary care, noting that the quality of IT systems across the NHS remains patchy, interoperability continues to be a challenge and progress towards digitisation of records is slow, made a series of recommendations to achieve digitisation.
• The DHSC’s 2018 ‘vision’ for digital, data and technology in health and care aspires to harness fast-developing technologies including AI and robotics, but also acknowledges the need to ‘get the basics right’ and ensure the NHS has appropriate digital infrastructure in place. This is essential not just for basic clinical and administrative functions, but also as a platform to enable the use of more sophisticated technologies.
• In 2018, NHS England and NHS Improvement launched the Local Health and Care Record Exemplars Programme, designed to enable the safe and secure sharing of health and care information across different parts of the NHS and social care.
• The NHS Long Term Plan, published in January 2019, also committed to making better use of digital technology, including providing better digital access to services and ensuring health records and care plans are available to clinicians and patients electronically. It also restated ambitions to make greater use of AI to support clinical decision making and to put in place IT infrastructure that is secure and allows interoperability between systems.
• NHSX, a joint unit between the DHSC and NHS England and NHS Improvement, created in 2019, provides leadership for digital technology in health and social care and will play an integral role in NHS England and NHS Improvement’s transformation directorate (see Box 7 on page 30 for further information). The 2019 Topol Review looked at how to equip the health care workforce to work effectively with new technologies, to inform the development of Health Education England’s (HEE’s) workforce strategy. HEE is seeking to address the workforce requirements set out in the Topol Review through the Digital Readiness Programme.

In Scotland, the Scottish government, the Convention of Scottish Local Authorities and NHS Scotland see digital technology as a critical enabler for improving health and care. The 2018 Digital Health and Care Strategy for Scotland seeks to empower citizens to manage their own health, live independently and access services through digital means, and also to put in place the architectural and information governance ‘building blocks’ necessary for the effective flow of information across the whole care system. Digital technology is also an important part of the Welsh government’s vision for health and care. A Healthier Wales: Our Plan for Health and Social Care seeks to use digital, data and communications technologies to help raise the quality and value of health and social care services. To help embed the development and use of digital services in health and care in Wales, the Welsh government launched a new special health authority, Digital Health and Care Wales, in April 2021.

The term ‘automation’ is not especially prominent in UK health care policy discourse, perhaps because it is often taken to mean the full automation of tasks (replacing human labour), whereas many of the technologies in question are seen primarily as tools to support staff to undertake tasks rather than to wholly automate them. Where automation has been discussed, the focus has been mainly on reducing the burden of administrative work for clinical staff, and improving efficiency and productivity, on the assumption that automation can free up time for patient care., Notwithstanding this lack of prominence of automation as a theme, there has been considerable policy interest in digital and data-driven technologies that have the potential to automate tasks, including AI:

• As part of the Industrial Strategy’s ‘AI Grand Challenge’, in 2019 the UK government set an ambition of using data, AI and innovation to transform the prevention, early diagnosis and treatment of diseases like cancer, diabetes, heart disease and dementia. This included launching five new centres of excellence in digital pathology and imaging with AI.
• In 2019, NHS England and NHS Improvement set an ambition for the NHS to become a world leader in AI and machine learning within five years, inviting technology innovators to submit proposals for ‘how the NHS can harness innovative solutions that can free up staff time and cut the time patients wait for results’.
• £250m was invested in the creation of a new NHS AI Lab, that sits within NHSX and focuses on areas such as regulation, imaging, disease detection, ethics and supporting the development of AI products. As part of this, the Artificial Intelligence in Health and Care Award is making £140m available over three years to accelerate the testing and evaluation of AI technologies that support the aims of the NHS Long Term Plan. The AI Lab has also launched an ethics initiative to ensure that AI products used in the NHS and care settings do not exacerbate health inequalities, in partnership with the Health Foundation, the National Institute for Health and Research, the Ada Lovelace Institute and HEE.
• The Accelerated Access Collaborative, a partnership between government, industry and the NHS, was established in 2018 to identify promising technologies, including AI, that the NHS should prioritise for adoption.
• In 2019, the DHSC and NHS England and NHS Improvement launched a code of conduct (since updated to become the Guide to good practice for digital and data-driven health technologies) to guide the development and use of digital and data-driven technologies, designed to protect patient data and ensure only high-quality technologies are used by the NHS.
• Prior to the announcement of the new National Institute for Health Protection, Public Health England’s Strategy 2020 to 2025 had set an ambition to develop ‘personalised prevention’ of ill health and enhance the data and surveillance capabilities of the public health system using technology such as AI.
• In 2018, the UK government announced the creation of a new AI health research centre in Scotland. Based in Glasgow, the Industrial Centre for Artificial Intelligence Research in Digital Diagnostics (iCAIRD) is focused on the exploration of how AI could improve patient diagnosis.
• The Welsh government is currently supporting several AI projects, including the use of AI to detect harmful or potentially harmful incidents in real time for people affected by falls, people with dementia and people with cognitive impairments.
• In 2020, the House of Lords Liaison Committee on AI published the report AI in the UK: No Room for Complacency, which considers the UK government’s progress against the recommendations made by the Select Committee on AI in its 2018 report, AI in the UK: ready, willing and able?
• In 2021, the UK AI Council, an independent expert committee that provides advice to the UK government, published a road map to help the UK become one of the best places in the world to live with, work with and develop AI.

1-3 Public, patient and professional attitudes to automation and AI

Several themes recur in surveys exploring public views of automation and AI in health care. While access, speed and accuracy are often cited as potential benefits, many people clearly value human agency, interaction and judgement and don’t want to see them compromised or removed. For example:

• An international survey in 2016 found that, while there is some support for using AI and robotics to meet health care needs, people in the UK were more sceptical compared to other countries. For example, UK respondents were least willing to undergo ‘surgery performed by a robot’. UK respondents felt that the main advantages of AI and robotics in health care were quicker and easier access, and faster and more accurate diagnosis. However, the main disadvantages cited were inability to trust automated decision making, the belief that only humans can make the right decisions and the view that health care needs a ‘human touch’.
• A 2017 UK poll found that while many would be happy with AI playing a supportive role, there were concerns about the automation of work typically done by doctors and nurses. In particular, the poll showed that while many respondents would welcome the use of AI to help diagnose diseases, most did not think AI should be used for other tasks usually performed by doctors and nurses, such as suggesting treatment.,
• Another 2017 UK poll found that people were optimistic about the potential of the technology to improve the accuracy and speed of diagnosis and were also ‘happy with the idea of doctors and machines working together to provide a better service’. Their main concern was the prospect of human interaction being lost. For example, respondents cited the importance of human involvement in final diagnosis and treatment planning, which they felt should be reviewed, authorised and communicated by a human doctor.

Turning to the perceptions of health care professionals and managers on the prospects for, and impact of, automation, there is a mixture of optimism and scepticism. For example:

• In a 2018 US survey of radiologists exploring views about job security, respondents said that AI would make their job radically different in the next 10–20 years, but very few felt that it would make their roles obsolete., Most respondents were planning to learn about AI in relation to their jobs and a smaller majority were willing to help train an algorithm to do some of the tasks of a radiologist. In the UK, the Royal College of Radiologists has similarly taken a positive view of AI, welcoming the introduction of appropriately regulated technologies to enhance clinical practice, citing the potential to improve outcomes and efficiency, and release time for direct patient care and research.
• A 2018 survey of managers and clinicians working in NHS trusts, clinical commissioning groups and NHS England and NHS Improvement found that while senior managers showed enthusiasm about AI, clinicians were more cautious, emphasising the need for safeguards.
• There appears to be less professional caution around the prospect of automating administrative tasks compared to clinical tasks. For example, a recent UK survey of GPs showed that while the majority were sceptical about the potential for future technology to perform most primary care tasks as well as or better than humans, many were optimistic that in the near future technology would have the capacity to fully replace GPs in undertaking administrative duties related to patient documentation., The Royal College of General Practitioners has highlighted the automation of administrative tasks as one of four key areas where technology could be particularly beneficial.,

Figure 2: Public and NHS staff familiarity with automation and AI

In general, how much, if anything, have you heard, seen or read about automation and AI in health care (eg in the news, on social media, or from family, friends, colleagues, etc.)?

Our surveys also asked how positive or negative people felt about the use of automation and AI in health care – as a crude ‘temperature test’. In both the public and NHS staff surveys, more felt positive than negative, but opinion was closely balanced, with the public feeling more positive than negative by 40% to 37% and with NHS staff surveyed feeling more positive than negative by 40% to 36%.

There were some interesting differences underneath these headline figures. In the public survey, some groups were less positive about automation and AI in health care than others, including women, people with a health condition or disability and people with a carer. While men felt more positive than negative about the use of automation and AI by 48% to 33%, women felt more negative than positive by 41% to 33%. Similarly, people with a carer felt more negative than positive by 42% to 34%, as did those with a health condition or disability by 41% to 38%. Age is also sometimes highlighted as a factor affecting attitudes to technology, and there were some modest age differences within our results, with younger people (aged 18–34) feeling more positive than negative by 41% to 31%, while for older people (aged 55 or older) this margin was just 1 percentage point: 41% to 40%. More research is needed to understand why these differences exist, but they underline the importance of engaging with patients and the public in the development and deployment of automation and AI to co-design solutions, in order to help make sure these technologies work for everyone and that different preferences are taken into account.

Among the NHS staff surveyed, there were some differences by professional group – perhaps reflecting the different perspectives that different staff groups will have on what these technologies might mean for health care. For example, medical and dental staff surveyed felt more positive than negative about the use of automation and AI in health care by 43% to 36%, and nurses and midwives by 39% to 38%, but health care assistants felt more negative than positive, by 41% to 33%.

However, these differences were dwarfed by the impact of familiarity with the topic. Among the public, those who said they had heard, seen or read ‘a lot’ or ‘a fair amount’ about automation and AI in health care felt much more positive than negative about the use of these technologies, by 70% to 26%, while those who answered ‘not very much’ or ‘nothing at all’ felt more negative than positive, by 41% to 35%. A similar pattern was evident in the NHS staff survey: those who said they had heard, seen or read ‘a lot’ or ‘a fair amount’ felt much more positive than negative about the use of automation and AI in health care, by 71% to 21%, while those who answered ‘not very much’ or ‘nothing at all’ felt more negative than positive, by 40% to 32%. This suggests that helping to familiarise people with this topic could play an important role in shaping attitudes to this agenda in future.

Figure 3: Public and NHS staff attitudes to the use of automation and AI in health care

Overall, how positive or negative do you feel about the use of automation and AI in health care?

Having explored the current context for automation and AI, in the next chapter we look at some of the opportunities to apply these technologies in health care.

---

2-1 Types of task amenable to automation

There are different ways of analysing and understanding tasks when considering what kinds of work can be automated and what cannot.

One influential approach, by economist David Autor, focuses on the broad nature of tasks and views the degree of routine involved in a task as a key determinant of automatability. Autor categorised tasks based on two properties: routine versus non-routine, and manual versus cognitive. Routine manual tasks require what Autor calls a ‘methodical repetition of an unwavering procedure’, such as picking and sorting items and repetitive assembly – the kind of repetition that could be explicitly programmed and performed by machines. Non-routine manual tasks, on the other hand, such as caretaker work and truck driving, are considered to have more limited scope for automation in Autor’s model. Routine cognitive tasks include record keeping and repetitive customer services like bank clerk work, and these are thought to offer substantial room for automation. Finally, non-routine cognitive tasks, which include medical diagnosis, legal writing and sales, are not seen to be candidates for automation and are viewed as more likely to remain (at least partially) in the domain of human performance.

Autor’s analysis has shone a light on labour market trends such as the ‘disappearing middle’, whereby – contrary to earlier predictions – many manual, non-routine jobs have remained resistant to automation, while other ‘white collar’ jobs, such as clerical work, have been automated. In these cases, it has been routine work that has been susceptible to automation, whether manual or cognitive, while non-routine work has proved harder to automate.

Autor’s analysis has been influential over many years, though technological advancements have led to certain non-routine tasks becoming easier to automate – truck driving, for example. Due to these advancements, some argue that automation can be extended to any non-routine task that is not subject to any ‘engineering bottlenecks’, which we discuss in more detail in Chapter 3.

This way of looking at tasks suggests that automation will tend to have most impact when applied to tasks that are frequent or time consuming, important and repetitive (though there can sometimes be benefits to applying automation to infrequent tasks too – for example, when a drug is rarely prescribed – as it is often these types of task where mistakes are most likely to occur). Such repetitive, rule-based tasks frequently lend themselves to a form of automation known as robotic process automation (RPA), which uses software to enable transaction processing, data manipulation and communication across multiple IT systems. RPA has successfully been deployed in many sectors to process invoices, claims and payments, monitor error messages and respond to routine requests from customers and suppliers.3 While RPA is not widespread in health care, a number of NHS trusts have successfully used it to manage scheduling and other processes, and NHSX is exploring how RPA can be scaled in the NHS through its Digital Productivity Programme.

A second approach to considering which kinds of task are amenable to automation is to analyse the skills and knowledge they require. The Oxford study used the O*NET database’s ‘occupation features’ to analyse how predictions of a task’s automatability related to the different skills and knowledge required. O*NET is a database of hundreds of standardised descriptors of almost 1,000 occupations in the US economy, which includes information on the skills, knowledge, work activities and interests associated with these occupations. The researchers looked at which features were clearly predictive of automatability and also at where an increase in the presence of a particular feature led to an increase in predicted automatability.

Table 1 presents the five largest (positive) percentage differences in O*NET occupational features for tasks that were rated automatable, not automatable and partly automatable, when compared to the entire dataset of health care tasks. Health care tasks predicted as ‘not-automatable’ require 26.0% higher personnel and human resources knowledge, 17.0% higher education and training knowledge and 18.7% higher management and financial resources skills. By contrast, ‘automatable’ tasks require 24.4% higher clerical skills.

Table 2 presents the occupational features with the largest positive influence on predicted task automatability, given an increase of the particular feature. Telecommunications and clerical knowledge have the largest gradient, meaning that if a task requires more of this, then its automatability score will increase to a greater extent than increasing other features.,

So these analyses suggest there might be a range of tasks in health care, especially clerical tasks, that could be promising candidates for automation.

Table 1: Largest feature differences relative to dataset by automatable category

Automatable category	O*NET feature	Feature difference
Not automatable	Installation (skill)	+62.6%
	Building and Construction (knowledge)	+27.2%
	Personnel and Human Resources (knowledge)	+26.0%
	Management and Financial Resources (skill)	+18.7%
	Education and Training (knowledge)	+17.0%
Automatable	Clerical (knowledge)	+24.4%
	Customer and Personal Service (knowledge)	+13.2%
	Service Orientation (skill)	+5.0%
	Economics and Accounting (knowledge)	+4.7%
	Computers and Electronics (knowledge)	+4.5%
Partly automatable	Medicine and Dentistry (knowledge)	+30.8%
	Therapy and Counselling (knowledge)	+22.0%
	Psychology (knowledge)	+13.4%
	Clerical (knowledge)	+13.2%
	Biology (knowledge)	+12.1%

Source: Willis M, et al.

Table 2: Ten largest O*NET occupational feature gradients

O*NET feature	Feature gradient
Telecommunications (knowledge)	+0.167
Clerical (knowledge)	+0.166
Wrist-finger Speed (ability)	+0.153
Number Facility (ability)	+0.118
Mathematics (skill)	+0.093
Depth Perception (ability)	+0.092
Building and Construction (knowledge)	+0.09
Mathematical Reasoning (ability)	+0.088
Economics and Accounting (knowledge)	+0.085
Control Precision (ability)	+0.082

Source: Willis M, et al.3

A third approach to thinking about the types of work that can be automated is to look at different task ‘functions’ and consider what level of automation might be appropriate in each case. To take one example, drawing on the different stages of human information processing, Parasuraman and colleagues describe four system functions that can potentially be automated: information acquisition, information analysis, decision and action selection, and action implementation. For each type of function there are different possible levels of automation. For example, with regard to decision making, a computer could suggest options to support human decision making, on the one hand, or it could actually make a decision and act autonomously, on the other; and there may be further levels in between, such as having a computer make the decision but still with the ability of a human operator to override it if necessary. Parasuraman and colleagues argue that a variety of factors will be relevant to determining the appropriate level of automation in different cases, especially the reliability of the automation technology, the potential impact on human performance in the resulting system and, crucially, the risks associated with decisions. So, for example, looking at air traffic control systems, they suggest that in the case of information acquisition and analysis, high levels of automation could be appropriate, while in the case of decision selection and implementation high levels of automation would only be appropriate for low-risk situations.

Work in health care is often complex and carries particular risks, with different kinds of tasks and work environments compared to other industries, and different legal, governance and institutional requirements. These factors might make it hard to translate some applications of automation from other industries to health care. This is explored in more detail in Chapter 3. First, we take a look at some specific areas where automation can be, and is being, applied in health care.

2-2 Application to administrative tasks in health care

Here we use the term ‘administrative’ loosely to mean tasks involved in the management or organisation of health care services, rather than in the direct delivery of them. Some administrative tasks may sit separately from clinical work, such as managing finances and payroll, while others may be embedded within it, such as processing prescriptions and managing hospital bed capacity.

Given that much administrative work involves routine information processing, the discussion above suggests that there may be significant opportunities for the automation of administrative tasks in health care. And with a high volume of administrative work in the NHS, it is easy to see why there is such appetite to take advantage of these opportunities, with the NHS Long Term Plan and NHSX both highlighting the potential of technology for reducing the burden of administrative work. Some administrative processes in health care are already being automated in particular settings, such as appointment booking and scanning letters in GP practices. However, advances in data availability, computational power and machine learning are creating possibilities for further automation.

The Oxford study estimated that around 44% of administrative tasks in general practice are ‘mostly’ or ‘completely’ automatable using current technology, based on its model derived from the opinions of automation experts (though, notably, the study also found that no single full-time role in general practice could be entirely automated). The research suggested that tasks such as managing finances and payroll, checking and sorting post, printing letters, texting patients, note-taking and letter scanning could all theoretically be automated. Given that such tasks require a considerable amount of time in any GP practice (including the time of GPs themselves), automating them could have a significant, positive impact in reducing the administrative burden and freeing up staff to focus on other work, particularly in light of current financial and workforce pressures.

Automation could make a particular difference with information-processing tasks that are time consuming and which have an important bearing on resources and patient care, such as in the management, scheduling and planning of clinical services. For example, in order to address high outpatient non-attendance rates, East Suffolk and North Essex NHS Foundation Trust has automated patient appointment reminders and cancellations using a simple text messaging process, reallocating free bookings when cancellations are made. The Trust claims that in 2018 this prevented 1,356 appointments being missed over a period of eight weeks, equivalent to a value of £216,960. Given the issue of non-attendance is common across health care services, this type of application could be useful in a range of other settings beyond secondary care, such as primary, community and mental health services.

Appointments and scheduling are also an area where AI and predictive analytics could add an extra dimension to automation. For example, Beth Israel Deaconess Medical Center in Boston is using the automated analysis of historical data to predict how much time a patient might need in the operating theatre and then using this to inform scheduling. AI can also help with the management of resources such as hospital beds. For example, funded by the Health Foundation, Chelsea and Westminster NHS Foundation Trust has developed a model to predict rises in acute hospital bed occupancy (discussed in more detail in Chapter 3), which could be deployed to provide an early warning system that gives staff sufficient time to avert occupancy crises.

Advances in natural language processing (NLP), which can extract structured (machine analysable) data from unstructured narrative texts (such as clinical notes and letters), mean that automation technologies have increasing potential to support text-based administrative work, for example processing and producing letters – activities that require a significant amount of time across the NHS. In general practice, the Oxford study found that letter work – such as opening letters, triaging, scanning and redirecting them to relevant staff members, and responding to them in different ways – entails a large amount of work, which is typically undertaken by receptionists and secretaries, but also often by GPs. The study also found that a significant amount of time is spent trawling through letters to find small amounts of relevant information. Automation technologies using NLP could therefore be used to scan for relevant information and present it to health care professionals, prioritising information by context and urgency, as well as to produce letters automatically as patients have their examinations. In addition to helping make letter management in the NHS more efficient and less time consuming, such approaches could also open up opportunities for a wider range of communications options for patients, tailored to their needs and preferences, such as emails, texts, large print, braille, audio recordings and language translations. Another way NLP can be of particular value is in analysing and categorising patient feedback, which can then be used to target quality improvement work.

Given AI-driven advances in speech recognition, there is also significant potential for the automation of speech-related administrative tasks. For example, speech recognition could be used to streamline documentation tasks by using NLP to analyse patient–clinician conversations and create notes, turning a discussion into a text transcript, summarising and annotating it to provide clinically relevant information and categorising it to appropriate sections of the medical record. After the consultation, the clinician could review a summary for editing and saving (though see Chapter 3 on potential challenges with the automation of note-taking). An application such as this could also make connections with information already in the patient’s record, or with analysis of similar patients, to support clinical decision making. For example, Nuance Communications and Microsoft have developed a system that records each patient interaction and uses AI to convert it to text, which is then saved into the electronic health record.

2-3 Application in clinical services

Beyond some of the established uses of automation in health care highlighted in Chapter 1, advances in computing, the codification of clinical knowledge and the availability of large datasets are creating scope for further automation of clinical tasks, or at least components of clinical tasks, especially those that relate to data analysis and decision selection. There is particular interest in the potential of data analytics and AI to support clinical decision making around diagnosis and treatment, where – as discussed above – the spectrum of automation could range from clinical decision support systems that assist human decision making all the way to scenarios in which decisions could be delegated to machines.

In many cases, these technologies are still in the early stages of development, testing and adoption, and the excitement about their potential currently outstrips the reality on the front line. Often, more real-world testing is required to investigate how technologies that have shown potential in the lab can be best deployed. For example, recent systematic reviews of studies comparing the diagnostic accuracy of AI-driven medical image analysis with that of health care professionals found that while many AI models demonstrated comparable accuracy, few studies were prospective or randomised trials in live clinical settings, nor did many present externally validated results. Nevertheless, there are a growing number of examples of automation and AI that have shown their potential, ranging along the whole patient pathway – from promotion and prevention, through diagnosis and treatment, all the way to rehabilitation and supporting people to live with long-term conditions. We highlight a few examples below.

Diagnosis: While AI has been applied to analyse a range of diagnostic data, some of the most impressive advances in recent years relate to diagnostic imaging. Machine learning potentially enables a high degree of accuracy in pattern recognition and the classification of images, with the ability to identify abnormalities, in some cases more acutely than the human eye. The application of machine learning in the analysis of medical scans and pathology slides therefore holds significant potential for supporting and improving the detection of diseases. Significant progress has been made in radiology, with AI being used in screening for conditions such as breast cancer. Studies show AI-powered screening systems can be a useful addition to the breast screening pathway; for example, one study by McKinney and colleagues showed that, while only tested using retrospective data, an AI system using deep learning improved specificity and sensitivity in predicting the development of malignancy when compared to a first clinical assessor and performed no worse in comparison to a second clinical assessor, including a reduction in the incidence of false positives and false negatives.

Risk assessment and prediction: AI can also assist clinical decisions by using data to assess risks and make predictions – for example, predicting if a patient’s condition is likely to deteriorate. For example, a project funded by the Health Foundation and led by King’s College London to improve outcomes in stroke care (described in more detail in Box 6) has developed a machine learning model that could be used to predict which patients are at highest risk of mortality after stroke. At a population level, AI could also be used for risk stratification and public health surveillance, such as for disease outbreak prediction and surveillance, something that could be particularly helpful in anticipating future waves of COVID-19 as well as the spread of other diseases.

Triage: There is also increasing interest in the potential of AI to help clinicians make triage decisions and to help patients assess their own symptoms. Several AI triage tools are currently being tested, although these are currently intended to support clinicians in decision making, rather than to make triage decisions themselves. For example, a Health Foundation-funded project led by Barking, Havering and Redbridge University Hospitals NHS Trust is developing a system to assist the emergency department triage process by quickly identifying high-risk patients needing urgent care, described in more detail in Box 13 (page 46).

Patient-facing symptom checkers: A number of companies have also developed patient-facing symptom checkers that are currently available in the UK. These are ‘chatbots’ that use AI to compare data gathered from the patient with medical knowledge, with the aim of helping people get a better understanding of when to seek medical attention and connecting them to the appropriate service. For example, Your.MD and Babylon Health have both developed symptom checkers that ask users a series of questions to build a picture of their symptoms, before suggesting the most appropriate course of action, such as recommending a visit to the GP or hospital or providing reassurance that the person can take steps to recover at home. Babylon claims its chatbot has been trained to recognise the ‘vast majority of health care issues seen in primary care’, although as of May 2021 it also says it is not suitable for people with mental health concerns, skin problems, and pregnancy or post-natal concerns, among others, recommending an appointment with a doctor., University Hospitals Birmingham NHS Foundation Trust is now working with Babylon to use this technology to develop a pre-hospital triage app for people considering using A&E services, as part of a drive to reduce unnecessary A&E attendances.

Treatment decisions and planning: As well as diagnosis and triage, data tools and AI can also be used to support better treatment decisions and planning, by suggesting treatment options or creating prompts and reminders to optimise the delivery of treatment. AI can be coupled with a range of patient data, including genomics data, as well as research and best practice, to help tailor treatments to individual patients. Electronic prescribing systems, for example, can incorporate decision support tools to check dosages and potential interactions with other drugs or conditions, and suggest safer or better-value alternatives. Used in this way, decision support systems – while stopping well short of complete automation – can be a useful tool in promoting adherence to best practice and helping reduce errors and unwarranted variation. In the case of electronic prescribing systems, for example, studies suggest that when used effectively these systems can reduce medication errors and adverse drug events,, although there is more work required to ensure they are consistently safe and effective.,

Health promotion and self-management: Automation and AI are also increasingly supporting people to manage their own health. For example, a range of technologies, such as apps, wearables and medical devices, are enabling health monitoring and management in homes and residential care settings. One example is the latest generation of insulin pumps, which combine continuous glucose monitors with smart algorithms that automatically adjust dosage.

Quality improvement: These technologies can also play an important role in supporting quality improvement by helping to generate data and analysis about how services are performing and how they can be improved – critical components of a learning health system. The projects at Imperial College Healthcare NHS Trust and King’s College London described above, which are using machine learning to analyse patient feedback and clinical audit data, demonstrate how AI can be used to help identify where improvements can be made.

In many of the areas highlighted above, robotics can be combined with AI to automate tasks that require movement as well as information processing. The example of surgery has already been highlighted. Another area where this has potential is medication dispensing and stock management in hospitals and community pharmacies; robotic dispensers can operate with greater speed and precision than humans, with a significantly lower risk of error., A further application of robotics in clinical settings is transportation. For example, Moxi is a robotic assistant with a mobile base, arm and gripper, reportedly being trialled in the US, designed to assist clinical staff by transporting supplies to patient rooms and delivering lab samples. Robotic technologies also present opportunities to provide therapy, whether in the home or in a variety of care settings. For example, Paro is a therapeutic AI robot, designed to look like a seal, to support people with dementia, which has shown signs of being able to reduce agitation and improve verbal and visual engagement among users.

While at present several of the technologies reviewed here currently require human supervision or input, and so are not examples of full automation, they could conceivably become more independent in future. Either way, used effectively, they have the potential to improve outcomes, experience and efficiency, as well as to help alleviate workforce pressures.

Figure 4: Range of potential uses for AI in health and care

Source: NHSX, Artificial Intelligence: How to get it right, 2019

Figure 5: Views of NHS staff on the biggest opportunities for automation and AI in health care

Which, if any, of the following do you think represent the biggest opportunities for using automation and AI to improve health care?

There were a few differences in the pattern of results among different occupational groups, mainly that groups tended to pick options that were particularly relevant to their work. For example, ambulance staff gave triage tools a higher rating than average (29% compared to 17% for all respondents) while public health professionals gave them a lower rating than average (13%). But the ranking of opportunities was broadly similar across occupational groups.

The list of opportunities provided was certainly not comprehensive, and it is worth noting in this respect that 6% of respondents thought that none of these listed opportunities were in their top three biggest opportunities (represented by the bar labelled ‘none of these’ in the figure above).

---

There is clearly great potential for the use of automation and AI in health care. But there are also some important constraints on where and how these technologies can be applied, along with a range of design and implementation challenges if they are to be successfully deployed. It is perhaps understandable that the promise of automation and AI typically excites the most attention. But given that the introduction of technologies into health care settings necessarily creates new risks and potential points of failure, we believe the challenges require just as much focus if we are to get automation right.

In this chapter, we explore some of these challenges and constraints by exploring different characteristics of tasks in health care. One set of challenges emerges from tasks requiring uniquely human traits or human presence. Another stems from the complexity of tasks and work environments in health care. We also explore the challenges of implementing and using automation technologies effectively in practice. While some of these issues might pose absolute constraints on the use of automation and AI in health care, others can be addressed through effective design and implementation.

We do not consider here challenges that relate specifically to the technologies themselves (such as interoperability with other systems) or the data they rely on (such as data protection), which are beyond the scope of this report – though Box 9 briefly highlights some important challenges related to machine learning. Rather, our focus is on the application and use of automation and AI in health care.

3-1 The human dimension

Many tasks in health care require human traits that cannot (yet) be replicated by machines. For other tasks, there is an intrinsic value of human agency or relationships meaning the task cannot be delegated to a machine. Both create limitations on the scope for automation.

3.1.1. Human traits that technology cannot (yet) replicate

Frey and Osborne’s influential work on the future of employment highlights three types of human ability that are difficult for technology to replicate: perception and manipulation; creativity; and emotional and social intelligence.

• Perception and manipulation: Replicating humans’ ability to understand and respond to external stimuli remains a challenge. Consider the use of a robot to assist a hospital patient to move from a bed to a toilet, or to clean the kitchen for someone needing help at home. While on the face of it these tasks might appear simple, they involve a vast number of distinct movements and the ability to respond to external stimuli in order to navigate through changing environments. Every possible movement and response requires codification; as Brynjolfsson and McAfee note, ‘low-level sensorimotor skills require enormous computational resources’. While machines can increasingly copy many aspects of human perception and movement, the ability to interpret the world and act appropriately remains much harder.
• Creativity: The psychological processes and values underlying human creativity are difficult to specify and therefore hard to automate. Creativity involves generating new ideas, or making new connections between familiar ideas. While this is not hard in itself, the issue is that creativity requires not just novelty but also value – which novel ideas or connections ‘make sense’? The challenge here lies in describing our creative values in a clear enough way for them to be programmed into a system., Later we discuss one example of where the need for creativity may constrain the scope of automation: in the strategies health care workers use to improvise and adapt when faced with unpredictable events.
• Emotional and social intelligence: Emotional and social intelligence, which are required in activities such as caregiving and managing people, pose challenges for automation. They include the ability to ‘read’ and deal effectively with the feelings of other people and to manage relationships. Automating this type of intelligence is a challenge not only because of the difficulty of recognising human emotion in real time, but also of knowing how to respond in an appropriate way, which requires a complex combination of skills and knowledge. Box 10 explores the role of emotional intelligence in health care.

3.1.2. Tasks requiring human presence

The use of automation in health care will not be determined simply by whether it is technically possible. We also need to understand where it is desirable, and where human input should be retained.

One key issue is the importance of human agency in care giving, particularly for treating patients with dignity and respect. While some patient-facing applications of automation, such as automated online appointment booking, might be considered perfectly compatible with treating patients with dignity and respect, others, such as robotic carers, are more contentious. A recent survey of 4,000 US adults on attitudes towards in-home robotic carers found that more were worried about the idea (47%) than enthusiastic (44%). The loss of human interaction was the predominant theme mentioned by respondents who said they wouldn’t want a robotic carer. In cases like these, people may feel that interacting with another human is necessary for, and indeed constitutive of, being treated with dignity and respect. There are a range of tasks in health care – such as informing a patient of a diagnosis of a serious illness – which for similar reasons may simply not be ‘delegable’ to a machine. As Liu and colleagues note, this type of communication requires ‘considerate assessment of a patient’s hopes, fears and expectations’, much of which is non-verbal and happens at an ‘innate level’, which an algorithm cannot replicate.

Another fundamental issue, as Batalden has observed, is that health care is not a product but a service that is co-produced with patients and families. Human relationships are fundamental for this co-production and shared decision making, which lie at the heart of person-centred care. In areas like care planning, for example, the need for genuine partnership between health care professionals and patients may pose ‘hard’ constraints on the use of automation.,

On other occasions it may not be that the use of automation is incompatible with dignity and respect, but simply that there is a preference for human contact. For example, in the Oxford study’s fieldwork, the researchers observed patients bypassing the touch-screen check-in at the GP surgery and choosing to check in with (and say hello to) the receptionist instead.

There are also a range of concerns that arise where automation and AI are used in decision making.,

Relating to the decision process are important issues of procedural fairness and acceptability concerning how decisions are made and resources allocated, where the increased use of automation and AI in decision making could have important ramifications. A significant finding of social psychology in recent decades is that people can care about the processes of decision making independently of the outcomes. For example, people who lose court cases may nevertheless feel more satisfied if they believe they’ve had a fair chance to have their argument heard compared to those who feel they haven’t. If automation and AI are used in diagnosis, triage and treatment decisions, interesting questions arise as to whether patients will feel these processes are fair and acceptable. For example, some patients might feel that algorithmic triage is acceptable as a process due to its potential to be consistent and free from certain kinds of human bias (though note the potential for data bias described in Box 9); whereas others might feel that having a human listen to and consider their case is a key component of being treated fairly and respectfully. More research is needed to understand perceptions of procedural fairness and acceptability surrounding automated decision making in health care; it is an area where the distinction between full and partial automation, and perceptions of the extent of machine involvement in decision making, may assume particular significance.

Where tasks have a direct impact on patient health, human participation in decision making may also be necessary for reasons of safety and accountability. For while automation can reduce the scope for human error, human oversight is often necessary to guard against machine error; in this sense, automated systems always require a ‘wrapper of human control’. Furthermore, the fact that machines can learn, make decisions and potentially act with a degree of autonomy raises important questions of accountability when mistakes occur. While attributing responsibility for problems can be complex (for example, when are they the responsibility of the machine operator and when of the manufacturer?), there is nevertheless general consensus that accountability must lie with humans rather than machines.

3-2 The complexity of work in health care

Another potential challenge for automation is the sheer complexity of tasks and work environments in health care – a prominent theme of sociological, ethnographic and human factors studies. Very often, there is more going on in a task than meets the eye. Attempts to introduce automation and reorganise work based on a simple interpretation of the task will therefore lead to problems or ‘unintended consequences’ – with potential risks to quality and safety.

Here we briefly discuss three potential challenges for automation arising from the complexity of work in health care: task multidimensionality (the fact that single tasks can fulfil multiple functions), task variation (the fact that tasks can look very different on different occasions and in different settings) and task unpredictability (the fact that tasks can evolve in ways that require flexibility and adaptation).

3.2.1. Task multidimensionality

One characteristic of many tasks in health care is their ‘multidimensionality’ – the fact that even in a single task there can be multiple things going on. Sometimes it might make sense to think about a task as comprising a range of ‘sub-tasks’ (in which case, the question of automation can move down a level to consider which sub-tasks can be automated). In other cases, it might make sense to think of the same task as serving multiple functions, and in these cases if you automate the task with the aim of fulfilling its ‘primary’ function, it will mean the other functions get lost.

The Oxford study identified a powerful example of this. While clinical documentation can be automated,,, which might well save time, the GPs in the study’s focus groups pointed out that this could remove an important opportunity for them to reflect on their cases. Such reflection is valuable not only for ensuring they make the best decisions about diagnosis and treatment, but also for reflecting on their practice more generally, which matters for professional development. Echoing Bansler’s observation that documentation can serve as a ‘tool of thinking’, the Oxford team observed how clinical notes were often kept open on screens for periods of time in order to enable GPs to consider, revise and make sense of the material, particularly notes for new or complex cases. Willis and Jarrahi comment that ‘If the practice of writing and thinking through writing is wholly removed from the clinicians’ workflow… it removes an opportunity for the clinician to think and reflect critically in the way they practice medicine.’ So the task of note-taking is potentially about more than recording information, and simply automating it and assuming this fulfils all the functions of the task would be wrong. Alternatively, clinical documentation could be automated but with GPs still taking time out to analyse and reflect on their cases in other ways. Some GPs may prefer this, though it wouldn’t necessarily result in the productivity gains that a simple interpretation of note-taking might lead one to expect.

In the same way that documentation can be about more than recording information, communication can be about more than transferring information. Ash and colleagues observe that communication can also be about generating an effect on the person you’re communicating with, testing their assumptions, receiving feedback, and establishing and maintaining relationships. Important functions like these can get lost if automated systems displace human communication and social interaction in the workplace.

Another kind of multidimensionality is where a decision-making task tacitly involves two different elements: the appraisal and decision making itself (decision selection); and the checking and ‘sign off’ of the decision (decision authorisation). When one person is performing the task, these two elements are usually elided. But while an alternative worker or computer might be able to perform the appraisal and decide on a course of action, this may not remove the need for a stage of checking and authorisation, and it may be undesirable to delegate this authority to the alternative worker or computer – particularly where the decision carries significant risk. So even when a decision-making task has been delegated or automated, there may still be a need for a suitably qualified worker to check the decision and sign it off. In some cases, this might still be more efficient than the previous arrangement; in other cases, the level of engagement required for the suitably qualified worker to familiarise themselves with the case and authorise the decision may be almost as much as if they were performing the whole task themselves, in which case the delegated arrangement may reduce productivity.

Studies of health care roles, including literature on ‘hidden work’, identify other examples of tasks fulfilling multiple functions. Health care assistants, for example, typically perform a range of tasks, including taking blood samples, monitoring vital signs, serving meals and helping patients move around. However, through the contact they have with patients in carrying out these tasks they also provide emotional support and identify patient needs (for example, needs around pain management), something that enables them to act as advocates for patients and bridge the relationship between patients and other clinicians. While it is conceivable that in future robots and automated systems may be used to assist patients with meals and mobilisation, and take measurements and samples, this could weaken an important dimension of patient support if it significantly reduced contact between health care workers and patients.

While none of the issues highlighted here are absolute barriers to automation, it is clearly important that proposals for automation are grounded in a detailed understanding of the work in question. What the literature on ‘unintended consequences’ reveals is that changes are sometimes made without this understanding.

3.2.2. Task variation

Many tasks and work activities in health care resist standardisation. This is not simply because patients and cases may differ, but also because roles, processes and workflows may be organised differently on different occasions and in different contexts, depending on the staff, skills and resources available. The Oxford study documents many types of variation across tasks in primary care (both within practices and between practices), including variation in the content of tasks (for example, phone calls), variation in the process for completing tasks (for example, handling correspondence) and variation in who performs tasks (for example, letter writing). According to the authors, ‘variance in tasks can occur in the order parts of the task are performed, duration, the occupational role of the person performing the task, the importance of the task or how time-critical it is, and how many individuals become involved in completing it’.

That tasks can be organised in different ways on different occasions may pose challenges for automation. For example, if the performance of a task is distributed between workers in different ways in different settings, then some task and work reorganisation may be necessary to make automation possible. For administrative tasks in primary care, the Oxford study found that the extent of task sharing – and therefore the way tasks are organised – varies depending on practice size, with more sharing of tasks at single-site practices than at large, multi-site practices.

In other cases, variation in who performs a task may reflect not simply variation in how the roles, processes and workflows are organised, but variation in the underlying nature or content of the task. The Oxford researchers noted that what looks like an administrative task can suddenly get transformed into a clinical task requiring specialist medical knowledge – something they argue ‘is what makes work in health care different and exceptional when compared to other fields with similar task descriptions’. Reviewing prescriptions, for example, an administrative necessity, can reveal information requiring a decision by a pharmacist with specialist knowledge. Medical coding is another type of task that can flip between administrative and clinical. Such variation in the nature of a task may render it less tractable to automation, or suggest only automating some aspects of it but not others.

Ultimately, variation itself need not be a challenge for automation provided that the nature of the variation is understood. But there are cases in health care where the parameters within which a task might vary can’t themselves be specified or constrained – the ‘left-field’ piece of information from a patient, for example, that could require a totally different approach to handling their case. Much automation discourse is grounded in the paradigm of product manufacturing, where tasks might be more prone to standardisation and routinisation than tasks in health care.119 So caution will be needed in assessing the applicability of the wider automation literature to health care.

3.2.3. Task unpredictability

One important source of task variation in health care is unpredictability in the way tasks unfold over time. This is partly because much health care work is responsive in nature – to the needs of patients, staff and organisations – and takes place in a dynamic work environment that can create disruptions to workflow. It is also because tasks are often being carried out in non-ideal circumstances where workers have to navigate uncertainty and trade off conflicting goals (such as whether to spend more time with one patient or move on to the next). While some disruptions result from operational failures (such as the need to fetch missing equipment) that can in principle be prevented, others result from intrinsic aspects of health care (such as the need to respond to a change in a patient’s condition).

Work interruptions are one example of this phenomenon that has been extensively studied. They pose challenges for task performance,, but while in some cases they are irrelevant to the task underway, in others they are vital for the delivery of safe care, requiring an immediate response.,

Managing fluid and unpredictable workflows safely requires flexibility, adaptation and improvisation from health care workers. But while automation technologies can potentially handle unpredictability, they can only do so if the appropriate strategies for handling it are understood. Ebright and colleagues highlight (in the context of nursing work) that not enough is known about the strategies and reasoning that front-line health care workers use to cope and adapt in complex work situations to be able to codify this knowledge. As Autor puts it, ‘The tasks that have proved most vexing to automate are those demanding flexibility, judgement, and common sense – skills that we understand only tacitly.’

While no one is proposing to automate complex clinical work such as nursing work, there is a wider challenge for automation here: the use of technologies in clinical pathways will need to support health care workers’ ability to improvise and adapt. One risk is if an automated system introduces unnecessary rigidities into the workflow, undermining flexibility and the capability to react. An example would be where an automated system won’t allow progression to the next stage of a task until certain information has been entered, despite the fact that real-life situations sometimes demand that task steps are initiated early, in the absence of the information or in an unconventional order. Another risk is where reliance on automation technologies restricts communication or takes workers ‘out of the loop’, reducing the situational awareness needed to respond to unexpected events.,

So automation technologies need to be designed in ways that match the realities of complex workflows and hectic work environments (discussed further in Box 11). Coiera observes that technology is often designed on the incorrect assumption of a fully concentrating user in a single-task scenario; the reality is that health care workers may be carrying out several tasks simultaneously or interacting with colleagues to help them complete other tasks. Ash highlights a range of problematic design features of technologies in this respect, such as interfaces that are hard to use in an interruptive context or that unnecessarily increase the cognitive load on staff by requiring information to be entered in overly structured formats.

In summary, while automation is best suited to routine work, work patterns on the front line are often adaptive and emergent as workers juggle competing demands under resource constraints. The successful design and deployment of automation technologies will need to support the ability of health care workers to flex and adapt in the face of task unpredictability.

Figure 6: Interrelationship between technology, process redesign and workforce redesign

 

Note: This model is based on an approach developed by Tan Tock Seng Hospital, Singapore

Source: Healthcare Improvement Scotland. Discussion Paper – an evolving approach to supporting the redesign and continuous improvement of health and care in Scotland. Healthcare Improvement Scotland; 2019

3-3 Challenges for implementing automation and AI in health care

Even when automation systems are designed in appropriate ways for live health care settings, there still remain significant challenges in implementing them effectively. If the success of a technological intervention depends not just on the technology itself but also on a whole set of accompanying role, process and workflow elements, and if these role, process and workflow elements can vary from one setting to another, then implementation can be viewed as the practice of fitting the technology into the specific organisational context. And this can be a complex process. As the 2016 Wachter Review of health IT in England argued, implementing digital technologies is not a simple case of ‘technical change’ (like following a recipe) but of highly complex, adaptive change, which requires substantive and long-lasting engagement between those leading change and those on the front line responsible for making technologies work.

The challenges of implementing automation technologies go beyond the usual set of challenges faced in implementing new health care interventions. First, there are specific challenges associated with the implementation and use of new technologies. Second, as highlighted in the literature on skill-mix change, there are specific challenges associated with ‘task shifting’ – work reorganisation that relocates the performance of a task. Automation potentially combines both technology and task shifting, making successful implementation far from straightforward.

As with many of the issues examined in the previous section, these kinds of challenge are not necessarily barriers to automation, but rather factors that will have to be addressed if automation is to work effectively. Failure to do so can lead to unintended consequences, such as operational failures, workarounds, decreased productivity and increased workloads.

3.3.1. Challenges related to technology implementation

A first set of challenges relates to the implementation of technologies. Here we highlight challenges related to the ‘human’ (sociotechnical) dimension of technological interventions; challenges relating specifically to the nature of technologies themselves or the data they rely on are beyond the scope of this report.

• Embedding new technology successfully in health care settings requires developing new organisational routines, ways of working and behaviours. The challenges of doing this are a recurring theme of evaluations of health technology interventions. They include the need to establish the implications of the new technology for organisational processes and workflows, the need to agree and coordinate the new ways of working required, and then – even when all of this is known – the need to actually change behaviours and ways of working, which may be deeply entrenched. Problems will arise if the workflow models used in developing the technology don’t match real-life workflows. As noted earlier, the complexity of work in health care, where there can be significant gaps between ‘work-as-done’ and ‘work-as-imagined’, means there may need to be a stage of process and workflow mapping to understand existing work patterns and think through how automation could be successfully introduced. Problems will also arise if there are not appropriate strategies and protocols in place to ensure safety and continuity if things go wrong. Relevant here is the ‘Safety II’ approach, which focuses on purposefully enabling things to go right, rather than merely seeking to prevent failures, and on being proactive rather than reactive. Safety II also focuses on the importance of adaptation and flexibility to match the conditions of work, and the role of humans to ‘absorb variability’ and create resilience, which is essential if technology is to work safely and effectively.

• Training and re-training are a critical part of successful implementation, particularly in light of the increasing complexity of automation, AI and robotic technologies, and functionalities that are new and unfamiliar. On this issue, the 2019 Topol Review argued there will need to be an increase in digital literacy among health care professionals.,, Health care workers will not only need to understand how to use new technologies safely; they may also need to understand how the technology works, at least at a basic level, in order to interpret the outputs of the system correctly or to explain the system’s operation to others.
• There is a risk that automation and AI could create or widen health inequalities. In addition to the risks to health inequalities associated with data that were highlighted in Box 9, digital inclusion poses particular challenges given inequalities in access to technology and digital literacy, especially in cases requiring patient interaction with new technologies or with a digital interface. It has been estimated that 11.7 million people do not have the essential digital skills needed for day-to-day life in the UK and they are more likely to be older, poorer, live with disabilities and need health care services. Approaches that rely on expensive technologies purchased by patients, such as smartphone symptom checkers, can also present barriers to access. So steps to promote inclusion – for example, user training, or creating user interfaces to suit differing levels of digital literacy – will be important to ensure that the use of digital platforms and tools doesn’t exclude particular groups. This will likely require meaningful co-design with people from a range of demographic groups to ensure these technologies meet diverse needs. Furthermore, certain digital technologies may not work well for everyone and some people may prefer a non-digital option, so it is important that accessible and high-quality non-digital alternatives are available where appropriate. The Health Foundation is exploring some of these issues with the Ada Lovelace Institute, as part of a research project examining how the adoption of data-driven technologies and systems during COVID-19 may have affected health inequalities.

3.3.2. Challenges related to task shifting

A second set of implementation challenges relates to the fact that automation often involves shifting the performance of a task from a human to a computer. The literature on skill-mix change shows that removing a task from a health care worker who has traditionally performed it and relocating it elsewhere can be tricky, and automation can be looked at as a radical case of skill-mix change. Here we highlight some challenges involved in task shifting, which are relevant for the implementation of automation.

• New ways of working will only be effective if there is consensus around, and ownership of, the new working arrangements. Involving staff in identifying the problems to be tackled and co-designing the solutions can be important steps to achieving this, as well as increasing the likelihood that the changes made are the optimum ones. Conversely, particular challenges might arise if there is suspicion about the underlying motivation for work reorganisation – for example, if staff think it is driven by cost-cutting and therefore potentially a threat to care quality. There is also a risk that work reorganisation results in workers feeling their role is being devalued, or creates anxiety about job security, potentially having a negative impact on staff morale. So it is important that proposals for reorganisation address a recognised challenge and are accompanied by a broader vision of role development, and that the stated reasons for change resonate with the intrinsic motivations of health care staff to deliver high-quality care. The example given in Box 13 highlights the importance of gaining consensus for the successful development and implementation of AI.

• Clarity around roles and responsibilities is essential, especially if the task requires team coordination. Without this, task shifting can create confusion about the division of labour and professional responsibilities. This in turn can risk tensions over the ownership of tasks, lead to staff being deployed inappropriately and create inefficiencies – for example, duplication of efforts if work is not fully handed over as intended., So carefully articulated role and task definition is vital to create shared understanding regarding new working arrangements.
• Another challenge is to consider the implications of task shifting for workloads and job quality. While removing tasks from someone’s role can potentially free up time, if they are expected to use the time freed up for more demanding or stressful work (for example, overseeing more complex patient caseloads), this could result in increased burnout – unless support is provided to help them meet the demands of their new responsibilities. Another risk is if task shifting means health care workers have less patient interaction, which could lead to reduced job satisfaction. For example, Verghese observes that monitoring patients using technology can draw clinicians away from the bedside, reducing the opportunity to experience the professional satisfaction that can come from bedside examination.

3.3.3. Specific implementation considerations for automation

If embedding technology in care pathways is a challenge, and shifting tasks around is a challenge, and automation combines both of these, then it is reasonable to expect implementing automation to involve many of the considerations outlined above. Beyond this, some specific challenges arise in relation to automation, which we highlight here.

• One challenge is avoiding the loss of skills and confidence among staff who are no longer required to perform certain tasks frequently, and therefore lose opportunities to practise these skills, even though they may still be required in the future. This de-skilling is often inadvertent; workers can actively contribute to it as they naturally take advantage of and accommodate to innovations in their everyday work. Health care organisations can help to mitigate this risk through continuing professional development and providing relevant opportunities for workers to practise important skills.
• The handover problem is where reliance on an automated system reduces the ability of staff to know how and when to take back control in the event of system failure or other risks. This is partly related to the issue of de-skilling highlighted above, but is also a product of the loss of situational awareness by staff, and sometimes also a lack of agreed strategies for taking back control. For example, Wears, Cook and Perry describe the unexpected failure of an automated drug-dispensing unit in an emergency department, which, with no agreed protocol for dealing with such a system failure, gave rise to a serious patient safety risk. This type of problem can be mitigated with the right planning – anticipating vulnerabilities, scenario modelling to help teams imagine different potential eventualities and devising appropriate protocols for what to do when there is a system failure, rather than having to invent such protocols in the moment.
• A third challenge is avoiding automation bias, where humans place excessive faith in the decisions made by automated systems, uncritically following their recommendations. This can distort professional judgement – for example, studies show medics’ diagnostic accuracy falls when they are simultaneously presented with inaccurate computer-aided diagnoses. A related phenomenon is ‘automation complacency’, where those charged with monitoring automated systems fail to pick up on errors as a result of placing too much trust in those systems and failing to challenge their outputs sufficiently. Research suggests that humans monitoring automated systems can be particularly susceptible to this when they have other concurrent tasks. The requirements of properly overseeing automated systems should therefore be factored into assumptions about staff time ‘freed up’ through automation.

Figure 7: The NASSS framework

 

Source: Greenhalgh et al.

Figure 8: NHS staff views on the biggest challenges for making automation technologies work on the front line

Which one or two of the following do you think will be the biggest challenges for using automation and AI effectively in delivering health care?

The ranking of challenges was broadly similar across different occupational groups, with patient suspicion being the most highly ranked challenge for all staff groups. Health care assistants were slightly more likely than other staff groups to pick patient suspicion as a challenge (picked by 49%). Furthermore, nurses, midwives and health care assistants were slightly more likely to pick staff shortages/inadequate equipment as a challenge (picked by 43% of these groups).

These results highlight the importance of engaging with patients to ensure support for the use of new technologies – for example, through consultation and co-designing changes – as an integral part of the process of adoption and implementation. They also highlight the importance of ensuring sufficient staff capacity to support change and adequate equipment and infrastructure – issues that are discussed further in the final chapter.

Having reviewed some opportunities and challenges for automation and AI in health care, in the final chapter we consider the implications for the future of work and look at what will be required in order to get the automation agenda right in practice.

---

1. Cameron G, Alderwick H, Bowers A, Dixon J. Shaping Health Futures: Preparing for tomorrow’s possibilities today. The Health Foundation; 2019 (www.health.org.uk/publications/reports/shaping-health-futures).
2. Willis M, Duckworth P, Coulter A, Meyer ET, Osborne M. Qualitative and quantitative approach to assess the potential for automating administrative tasks in general practice. BMJ Open. 2020; 10:e032412.
3. Willis M, Duckworth P, Graham L, Osborne M, Meyer ET, Coulter A. The Future of Healthcare: Computerisation, Automation and General Practice Services. University of Oxford; 2020 (https://bit.ly/foh-report-jun2020).
4. Encyclopaedia Britannica [webpage]. Automation. Encyclopaedia Britannica; 2020 (www.britannica.com/technology/automation).
5. International Society of Automation. What is Automation? [webpage]. International Society of Automation; 2020 (www.isa.org/about-isa/what-is-automation).
6. McKinsey Global Institute. A Future that Works: Automation, Employment, and Productivity. McKinsey Global Institute; 2017 (www.mckinsey.com/featured-insights/digital-disruption/harnessing-automation-for-a-future-that-works).
7. Office for National Statistics. Which occupations are at highest risk of being automated? ONS; 25 March 2019 (www.ons.gov.uk/employmentandlabourmarket/peopleinwork/employmentandemployeetypes/articles/whichoccupationsareathighestriskofbeingautomated/2019-03-25).
8. IBM. Apply intelligent automation to your operations. IBM; 2020 (www.ibm.com/automation/automation-intelligence).
9. Stanford University. Artificial Intelligence Index 2017 Annual Report. Stanford University; 2017 (www.aiindex.org/2017-report.pdf).
10. House of Commons Science and Technology Committee. Robotics and artificial intelligence: Fifth Report of Session 2016–17. House of Commons; 2016 (https://publications.parliament.uk/pa/cm201617/cmselect/cmsctech/145/145.pdf).
11. Jarrahi M. Artificial intelligence and the future of work: human-ai symbiosis in organizational decision making. Business Horizons. 2018; 61(4).
12. Wang D, Khosla A, Gargeya R, Irshad H, Beck A. Deep learning for identifying metastatic breast cancer. arXiv. 2016 (https://arxiv.org/pdf/1606.05718.pdf).
13. Adidas’s high-tech factory brings production back to Germany. The Economist; 14 January 2017.
14. Oxford Economics. How robots change the world: What automation really means for jobs and productivity. Oxford Economics. 2019 (www.oxfordeconomics.com/recent-releases/how-robots-change-the-world).
15. MIT interview with David Mindell.
16. Dellot B, Wallace-Stephens F. The Age of Automation: Artificial intelligence, robotics and the future of low-skilled work. Royal Society for Arts, Manufactures and Commerce; 2017.
17. Brynjolfsson E, McAfee A. The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies. W.W. Norton & Company; 2014.
18. Novek J. Hospital pharmacy automation: Collective mobility or collective control? Soc Sci Med. 2000; 51(4):491–503.
19. Dahl T, Boulos M. Robots in health and social care: a complementary technology to home care and telehealthcare? Robotics. December 2013; 3(1): 1–21.
20. New surgical robots are about to enter the operating theatre. The Economist; 16 November 2017.
21. Harwich E, Laycock K. Thinking on its own: AI in the NHS. Reform; 2018.
22. Darzi A, Kibasi T, Quilter-Pinner H. Better health and care for all: A 10-point plan for the 2020s. Institute for Public Policy Research; 2018 (www.ippr.org/files/2018-06/better-health-and-care-for-all-june2018.pdf).
23. Department of Health. Operational productivity and performance in English NHS acute hospitals: Unwarranted variations. An independent report for the Department of Health by Lord Carter of Coles. Department of Health; 2016 (https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/499229/Operational_productivity_A.pdf).
24. Frey C, Osbourne M. The Future of Employment: How susceptible are jobs to computerisation? University of Oxford; 2013 (www.oxfordmartin.ox.ac.uk/downloads/academic/The_Future_of_Employment.pdf).
25. Arntz M, Gregory T, Zierahn U. The Risk of Automation for Jobs in OECD Countries: A Comparative Analysis, OECD Social, Employment and Migration Working Papers, No. 189. OECD Publishing; 2016 (https://dx.doi.org/10.1787/5jlz9h56dvq7-en).
26. Servoz M. AI: The future of work? Work of the future! European Commission; 2019 (https://ec.europa.eu/digital-single-market/en/news/future-work-work-future).
27. Hawksworth J, Berriman R, Goel S. Will robots really steal our jobs? An international analysis of the potential long-term impact of automation. PwC; 2018 (www.pwc.co.uk/economic-services/assets/international-impact-of-automation-feb-2018.pdf).
28. Department for Business, Energy & Industrial Strategy. Good Work: The Taylor Review of Modern Working Practices. Department for Business, Energy & Industrial Strategy; 2017 (https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/627671/good-work-taylor-review-modern-working-practices-rg.pdf).
29. Department of Health. Making IT work: harnessing the power of health information technology to improve care in England. Department of Health; 2016 (www.gov.uk/government/publications/using-information-technology-to-improve-the-nhs/making-it-work-harnessing-the-power-of-health-information-technology-to-improve-care-in-england).
30. Department of Health and Social Care. The future of healthcare: our vision for digital, data and technology in health and care. Department of Health and Social Care; 2018 (www.gov.uk/government/publications/the-future-of-healthcare-our-vision-for-digital-data-and-technology-in-health-and-care).
31. NHS England and Local Government Association. Local Health and Care Record Exemplars. NHS England and LGA; 2018 (www.england.nhs.uk/wp-content/uploads/2018/05/local-health-and-care-record-exemplars-summary.pdf).
32. NHS England. NHS Long Term Plan. NHS England; 2019 (www.longtermplan.nhs.uk/wp-content/uploads/2019/08/nhs-long-term-plan-version-1.2.pdf).
33. NHSX. Driving forward the digital transformation of health and social care. [webpage]. NHSX (www.nhsx.nhs.uk/).
34. NHS and Health Education England. The Topol Review: Preparing the healthcare workforce to deliver the digital future. An independent report on behalf of the Secretary of State for Health and Social Care. NHS and Health Education England; 2019 (https://topol.hee.nhs.uk/).
35. Health Education England. Digital Readiness Programme [webpage]. Health Education England (www.hee.nhs.uk/our-work/digital-readiness).
36. Scottish Government, Convention of Scottish Local Authorities and NHS Scotland. Scotland’s Digital Health and Care Strategy. Scottish Government, Convention of Scottish Local Authorities and NHS Scotland; 2018 (www.gov.scot/publications/scotlands-digital-health-care-strategy-enabling-connecting-empowering/).
37. Welsh Government. A Healthier Wales: Our Plan for Health and Social Care. Welsh Government; 2019 (https://gov.wales/sites/default/files/publications/2019-10/a-healthier-wales-action-plan.pdf).
38. NHS Wales. New NHS Wales organisation for digital, data and technology [webpage]. NHS Wales (https://dhcw.nhs.wales/news/latest-news/new-nhs-wales-organisation-for-digital-data-and-technology/).
39. Gould, M. NHSX: giving patients and staff the technology they need [webpage]. UK Government; 2019 (https://healthtech.blog.gov.uk/2019/06/24/nhsx-giving-patients-and-staff-the-technology-they-need/).
40. Department for Business, Energy & Industrial Strategy. The Grand Challenge missions [webpage]. Department for Business, Energy & Industrial Strategy; 2019 (www.gov.uk/government/publications/industrial-strategy-the-grand-challenges/missions#early-diagnosis).
41. NHS England. NHS aims to be a world leader in artificial intelligence and machine learning within 5 years [webpage]. NHS England; 2019 (www.england.nhs.uk/2019/06/nhs-aims-to-be-a-world-leader-in-ai-and-machine-learning-within-5-years/).
42. NHSX. The NHS AI Lab (www.nhsx.nhs.uk/ai-lab).
43. NHS England. Artificial Intelligence in Health and Care Award [webpage]. NHS England; 2020 (www.england.nhs.uk/aac/what-we-do/how-can-the-aac-help-me/ai-award/).
44. NHSX. The AI Ethics Initiative [webpage]. NHSX (www.nhsx.nhs.uk/ai-lab/ai-lab-programmes/ethics/).
45. NHS England. NHS Accelerated Access Collaborative [webpage]. NHS England (www.england.nhs.uk/aac/)
46. Department of Health and Social Care. Code of conduct for data-driven health and care technology. Department of Health and Social Care; 2019 (www.gov.uk/government/publications/code-of-conduct-for-data-driven-health-and-care-technology/initial-code-of-conduct-for-data-driven-health-and-care-technology).
47. Public Health England. PHE Strategy 2020–25. PHE; 2019 (www.gov.uk/government/publications/phe-strategy-2020-to-2025).
48. The Industrial Centre for Artificial Intelligence Research in Digital Diagnostics. What is iCAIRD? [webpage]. The Industrial Centre for Artificial Intelligence Research in Digital Diagnostics; 2020 (https://icaird.com/about/).
49. Digital Health.Wales. Projects [webpage]. Welsh Government (https://digitalhealth.wales/projects).
50. House of Lords Liaison Committee. AI in the UK: No Room for Complacency. Seventh Report of Session 2019-21. House of Lords; 2020 (https://publications.parliament.uk/pa/ld5801/ldselect/ldliaison/196/196.pdf).
51. House of Lords Select Committee on AI. AI in the UK: ready, willing and able? Report of Session 2017-19. House of Lords; 2018 (https://publications.parliament.uk/pa/ld201719/ldselect/ldai/100/100.pdf).
52. UK AI Council. AI Roadmap. UK AI Council; January 2021 (https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/949539/AI_Council_AI_Roadmap.pdf).
53. PwC. What doctor? Why AI and robotics will define New Health. Survey results. PwC; 2016 (www.pwc.com/gx/en/industries/healthcare/publications/ai-robotics-new-health/survey-results.html).
54. Future Advocacy. Artificial Intelligence: YouGov Poll 2017. Future Advocacy; 2017.
55. Ipsos Mori Social Research Institute. Public views of Machine Learning: Findings from public research and engagement conducted on behalf of the Royal Society. Ipsos Mori; 2017 (https://royalsociety.org/-/media/policy/projects/machine-learning/publications/public-views-of-machine-learning-ipsos-mori.pdf).
56. Collado-Mesa F, Alvarez E, Arheart K. The role of artificial intelligence in diagnostic radiology: a survey at a single radiology residency training program. J Am Coll Radiol. 2018; 15:12, 1753–1757. doi: 10.1016/j.jacr.2017.12.021.
57. Royal College of Radiologists. RCR position statement on artificial intelligence [webpage]. Royal College of Radiologists; 2018 (www.rcr.ac.uk/posts/rcr-position-statement-artificial-intelligence).
58. Feary SR. Artificial Intelligence in healthcare – what does the NHS workforce think? Politics Home; 2018 (www.politicshome.com/thehouse/article/artificial-intelligence-in-healthcare--what-does-the-nhs-workforce-think).
59. Blease C, Bernstein MH, Gaab J, Kaptchuk TJ, Kossowsky J, Mandl KD, Davis RB, DesRoches CM. Computerization and the future of primary care: A survey of general practitioners in the UK. PLoS One. 2018; 13(12): e0207418 (https://doi.org/10.1371/journal.pone.0207418).
60. Royal College of General Practitioners. All Systems GP. Royal College of General Practitioners; 2019 (www.rcgp.org.uk/-/media/Files/News/2019/RCGP-tech-manifesto-april-2019.ashx?la=en).
61. Autor D, Levy F, Murnane RJ. The Skill Content of Recent Technological Change: An Empirical Exploration. The Quarterly Journal of Economics. 2003; 118(4): 1279–1333.
62. Davies A. Uber’s Self-Driving Truck Makes Its First Delivery: 50,000 Beers. Wired; 25 October 2016 (www.wired.com/2016/10/ubers-self-driving-truck-makes-first-delivery-50000-beers/).
63. Frey C, Osborne M. The future of employment: How susceptible are jobs to computerisation? Technological Forecasting & Social Change. 2017; 114: 254–280.
64. Deloitte. Robotic Process Automation [webpage]. Deloitte (www2.deloitte.com/uk/en/pages/innovation/solutions/robotic-process-automation.html).
65. Yorkshire & Humber AHSN. Exploring Robotic Process Automation in the NHS [webpage]. Yorkshire & Humber AHSN; 2021 (www.yhahsn.org.uk/blog/exploring-robotic-process-automation-in-the-nhs/).
66. Parasuraman R, Sheridan TB, Wickens CD. A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics – Part A: Systems and Humans. 2000; 30(3); 286–297. doi: 10.1109/3468.844354.
67. Thoughtonomy. East Suffolk and North Essex NHS Foundation Trust Case Study: Public Sector Healthcare. Thoughtonomy; 2018 (https://cdn2.hubspot.net/hubfs/4090201/2018-10%20October%20CONTENT/ESNEFT-%20Case%20Study.pdf).
68. Tozzi, J. Amazon Gives AI to Harvard Hospital in Tech’s Latest Health Push. Bloomberg; 4 March 2019 (www.bloomberg.com/news/articles/2019-03-04/amazon-gives-ai-to-harvard-hospital-in-tech-s-latest-health-push).
69. The Health Foundation. Listen, Learn and Improve: Language analysis of patient feedback [webpage]. The Health Foundation (www.health.org.uk/improvement-projects/listen-learn-and-improve-language-analysis-of-patient-feedback).
70. Pifer R. AI clinical documentation player Nuance preps for nationwide expansion. Healthcare Dive. 25 February 2020 (www.healthcaredive.com/news/ai-clinical-documentation-player-nuance-preps-for-nationwide-expansion/572847/).
71. Topol, EJ. High-performance medicine: the convergence of human and artificial intelligence. Nature Medicine. 2019. 25(1): 44–56.
72. Nagendran M, Chen Y, Lovejoy CA, Gordon AC, Komorowski M, Harvey H, Topol EJ, Ioannidis JPA, Collins GS, Maruthappu M. Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies. BMJ. 2020; 368: m689.
73. Liu X, Faes L, Kale AU, Wagner SK, Fu DJ, Bruynseels A, Mahendiran T, Moraes G, Shamdas M, Kern C, Ledsam JR, Schmid MK, Balaskas K, Topol EJ, Bachmann LM, Keane PA, Denniston AK. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digital Health. 2019; 1(6):271–297. doi: 10.1016/S2589-7500(19)30123-2.
74. Jiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, Wang Y, Dong Q, Shen H, Wang Y. Artificial intelligence in healthcare: past, present and future. Stroke and Vascular Neurology. 2017; 2: 230–243. doi: 10.1136/svn-2017-000101.
75. McKinney SM, Sieniek M, Godbole V, Godwin J, Antropova N, Ashrafian H, Back T, Chesus M, Corrado GS, Darzi A, Etemadi M, Garcia-Vicente F, Gilbert FJ, Halling-Brown M, Hassabis D, Jansen S, Karthikesalingam A, Kelly CJ, King D, Ledsam JR, Melnick D, Mostofi H, Peng L, Reicher JJ, Romera-Paredes B, Sidebottom R, Suleyman M, Tse D, Young KC, De Fauw J, Shetty S. International evaluation of an AI system for breast cancer screening. Nature. 2020; 577: 89–94. doi: 10.1038/s41586-019-1799-6.
76. Schwalbe N, Wahl B. Artificial intelligence and the future of global health. The Lancet. 2020; 395: 1579–1586.
77. The Health Foundation. Developing and implementing machine learning driven analytics for quality improvement in health care [webpage]. The Health Foundation (www.health.org.uk/research-projects/developing-and-implementing-machine-learning-driven-analytics-for-quality).
78. Babylon Health. Ask Babylon [webpage]. Babylon Health (www.babylonhealth.com/product/ask-babylon).
79. Babylon Health. When not to use the Babylon chatbot [webpage]. Babylon Health (https://support.babylonhealth.com/hc/en-us/articles/213450625-When-not-to-use-the-Babylon-chatbot).
80. Tallon L. 2019/20 Annual Plan Progress Update. University Hospitals Birmingham NHS Foundation Trust; 25 July 2019 (www.uhb.nhs.uk/Downloads/pdf/Bod0719AnnualPlanProgressUpdate.pdf).
81. Radley DC, Wasserman MR, Olsho LE, Shoemaker SJ, Spranca MD, Bradshaw B. Reduction in medication errors in hospitals due to adoption of computerized provider order entry systems. J Am Med Inform Assoc. May 2013; 20(3): 470–476.
82. Nuckols TK, Smith-Spangler C, Morton SC, Asch SM, Patel VM, Anderson LJ, Deichsel EL, Shekelle PG. The effectiveness of computerized order entry at reducing preventable adverse drug events and medication errors in hospital settings: a systematic review and meta-analysis. Systematic Reviews. 2014; 3:56. doi: 10.1186/2046-4053-3-56.
83. Schiff G, Mirica MM, Dhavle AA, Galanter WL, Lambert B, Wright A. A prescription for enhancing electronic prescribing safety. Health Affairs. 2018; 37(11): 1877–1883. doi: 10.1377/hlthaff.2018.0725.
84. Messer LH, Berget C, Forlenza GP. A Clinical Guide to Advanced Diabetes Devices and Closed-Loop Systems Using the CARES Paradigm. Diabetes Technology & Therapeutics. May 2019; 21(8).
85. Pickett, M. Robots Are Now Handling Pills. Will Pharmacists Be Liberated or Out of Work? KQED Science; 28 April 2016 (www.kqed.org/futureofyou/153628/when-a-robot-counts-out-your-pills-what-will-your-pharmacist-do).
86. How Moxi, helper robot from Diligent Robotics, has joined Covid-19 fight. Austin Business Journal. 21 December 2020. (www.bizjournals.com/austin/news/2020/12/21/how-moxi-helper-robot-from-diligent-robotics.html).
87. Moyle W, Jones CJ, Murfield JE, Thalib L, Beattie ERA, Shum DKH, O’Dwyer ST, Mervin MC, Draper BM. Use of a Robotic Seal as a Therapeutic Tool to Improve Dementia Symptoms: A Cluster-Randomized Controlled Trial. J Am Med Dir Assoc. 2017; 18(9):766–773. doi: 10.1016/j.jamda.2017.03.018.
88. Joshi I, Morley J (eds). Artificial Intelligence: How to get it right. Putting policy into practice for safe data-driven innovation in health and care. NHSX; 2019.
89. Chen JH, Asch SM. Machine Learning and Prediction in Medicine — Beyond the Peak of Inflated Expectations. New England Journal of Medicine. 2017; 376: 2507–2509. doi: 10.1056/NEJMp1702071.
90. Char D, Shah N, Magnus D. Implementing Machine Learning in Health Care — Addressing Ethical Challenges. New England Journal of Medicine. 2018; 378(11): 981–983.
91. Grimm F. Predicting missed hospital appointments using machine learning — what are the risks? Towards Data Science. 2 August 2019 (https://towardsdatascience.com/predicting-missed-hospital-appointments-using-machine-learning-what-are-the-risks-a388348109d).
92. Campolo A, Sanfilippo M, Whittaker M, Crawford K. AI Now 2017 Report. AI Now Research Institute, New York University; 2017.
93. Challen R, Denny J, Pitt M, Gompels L, Edwards T, Tsaneva-Ttanasova K. Artificial intelligence, bias and clinical safety. BMJ Quality & Safety. 2019; 28: 231–237.
94. Yeung, K. Why worry about decision making by machine? In: Yeung K, Lodge M (eds.) Algorithmic Regulation. Oxford University Press; 2019.
95. Goleman, D. Emotional Intelligence: Why It Can Matter More Than IQ. Bloomsbury Publishing; 1996.
96. Salovey P, Brackett MA, Mayer JD. Emotional intelligence: key readings on the Mayer and Salovey model. National Professional Resources; 2004.
97. Serrat O. Understanding and Developing Emotional Intelligence. In: Serrat O (ed.) Knowledge Solutions. Springer; 2017.
98. Stanton E, Noble D. Emotional intelligence. BMJ. 2010; 341: 3781.
99. Goleman D. What makes a leader? Harvard Business Review. January 2004 (https://hbr.org/2004/01/what-makes-a-leader).
100. The Health Foundation. Person-centred care made simple. The Health Foundation; 2014 (www.health.org.uk/publications/person-centred-care-made-simple).
101. Balik B, Conway J, Zipperer L, Watson J. Achieving an Exceptional Patient and Family Experience of Inpatient Hospital Care. IHI Innovation Series white paper. Institute for Healthcare Improvement; 2011.
102. Drake R. Five dilemmas associated with e-rostering. Nursing Times. 2014; 110(20): 14–16.
103. Smith A, Anderson M. Americans’ attitudes toward robot caregivers. In: Smith A, Anderson M. Automation in Everyday Life. Pew Research Center; 2017 (www.pewresearch.org/internet/2017/10/04/americans-attitudes-toward-robot-caregivers/).
104. Liu X, Keane PA, Denniston AK. Time to regenerate: the doctor in the age of artificial intelligence. Journal of the Royal Society of Medicine. 2018; 111(4): 113–116.
105. Batalden P. Getting more health from healthcare: quality improvement must acknowledge patient coproduction. BMJ. 2018; 362: 3617.
106. Frey BS, Benz M, Stutzer A. Introducing Procedural Utility: Not Only What, But Also How Matters. Journal of Institutional and Theoretical Economics. 2004; 160(3): 377–401.
107. Lind E, Tyler T. The Social Psychology of Procedural Justice. Springer; 1988.
108. Mindell D. Our Robots, Ourselves: Robotics and the Myths of Autonomy. Viking Press; 2015.
109. European Group on Ethics in Science and New Technologies. Statement on Artificial Intelligence, Robotics and ‘Autonomous’ Systems. European Commission; 2018 (https://ec.europa.eu/research/ege/pdf/ege_ai_statement_2018.pdf) .
110. Klann JG, Szolovits P. An intelligent listening framework for capturing encounter notes from a doctor-patient dialog. BMC Medical Informatics and Decision Making. November 2009; 9(1): S3.
111. Lin SY, Shanafelt TD, Asch SM. Reimagining clinical documentation with artificial intelligence. Mayo Clin. Proc. 2018; 93: 563–565.
112. Verghese A, Shah NH, Harrington RA. What this computer needs is a physician: humanism and artificial intelligence. JAMA. 2018; 319:19.
113. Bansler JP, Havn EC, Schmidt K, Mønsted T, Petersen HH, Svendsen JH. Cooperative epistemic work in medical practice: an analysis of physicians’ clinical notes. Computer Supported Cooperative Work. 2016; 25: 503–546. doi: 10.1007/s10606-016-9261-x.
114. Willis M, Jarrahi MH. Automating documentation: a critical perspective into the role of artificial intelligence in clinical documentation. In: Taylor NG, Christian-Lamb C, Martin MH, Nardi B (eds). Information in Contemporary Society: 14th International Conference. Springer; 2019.
115. Ash J, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc. 2004; 11(2): 104–112.
116. Allen, D. The invisible work of nurses: hospitals, organisation and healthcare. Routledge; 2015.
117. Royal College of Nursing. The role of the health care assistant [webpage]. Royal College of Nursing (https://rcni.com/hosted-content/rcn/first-steps/role-of-health-care-assistant).
118. Fernández-Macías E, Gómez E, Hernández-Orallo J, Loe BS, Martens B, Martínez-Plumed F, Tolan S. A multidisciplinary task-based perspective for evaluating the impact of AI autonomy and generality on the future of work. arXiv. 2018 (https://arxiv.org/abs/1807.02416).
119. Swinglehurst D, Greenhalgh T. Caring for the patient, caring for the record: An ethnographic study of ‘back office’ work in upholding quality of care in general practice. BMC Health Services Research. 2015; 15: 177.
120. Kohn LT, Corrigan JM, Donaldson MS (eds). To Err is Human: Building a Safer Health System. Institute of Medicine (US) Committee on Quality of Health Care in America. National Academies Press; 2000.
121. Chisholm CD, Dornfeld AM, Nelson DR, Cordell WH. Work interrupted: A comparison of workplace interruptions in emergency departments and primary care offices. Annals of Emergency Medicine. 2001; 38(2): 146–151.
122. Rivera-Rodriguez AJ, Karsh B. Interruptions and distractions in healthcare: review and reappraisal. BMJ Quality & Safety. 2010; 19: 304–312.
123. Sasangohar F, Donmez B, Trbovich P, Easty AC. Not all interruptions are created equal: positive interruptions in healthcare. Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 2012; 56(1): 824–828.
124. Ebright P, Patterson E, Chalko B, Render M. Understanding the complexity of registered nurse work in acute care settings. J Nurs Adm. 2003; 33(12): 630–638.
125. Autor D. Why are there still so many jobs? The history and future of workplace automation and anxiety. MIT IDE Research Brief. MIT Initiative on the Digital Economy. 2016; 07 (http://ide.mit.edu/sites/default/files/publications/IDE_Research_Brief_v07.pdf).
126. Wiener E, Curry R. Flight-deck automation: promises and problems. Ergonomics. 1980; 23: 995–1011.
127. Endsley MR, Kiris EO. The out-of-the-loop performance problem and level of control in automation. Human Factors. 1995; 37(2): 381–394.
128. Coiera E, Ash J, Berg M. The Unintended Consequences of Health Information Technology Revisited. Yearb Med Inform. 2016; 1: 163–169.
129. Coiera E. Guide to Health Informatics, third edition. CRC Press; 2015.
130. Dixon-Woods M, Martin G, Tarrant C, Bion J, Goeschel C, Pronovost P, Brewster L, Shaw L, Sutton L, Willars J, Ketley D, Woodcock T. Safer Clinical Systems: evaluation findings. The Health Foundation; 2014 (www.health.org.uk/publications/safer-clinical-systems-evaluation-findings).
131. Shorrock S. The Varieties of Human Work [webpage]. Humanistic Systems. 2016 (https://humanisticsystems.com/2016/12/05/the-varieties-of-human-work/).
132. Perry S, Wears R, Cook R. The role of automation in complex system failures. Journal of Patient Safety. 2005; 1(1): 56–61.
133. Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. BMJ Quality & Safety. 2010; 19: 68–74.
134. Novak LL, Holden RJ, Anders SH, Hong JY, Karsh BT. Using a sociotechnical framework to understand adaptations in health IT implementation. Int J Med Inform. 2013; 82(12): 331–44.
135. Heifetz R, Laurie D. The Work of Leadership. Harvard Business Review. December 2001 (https://hbr.org/2001/12/the-work-of-leadership).
136. Shaw S, Wherton J, Vijayaraghavan S, Morris J, Bhattacharya S, Hanson P, Campbell-Richards D, Ramoutar S, Collard A, Hodkinson I, Greenhalgh T. Advantages and limitations of virtual online consultations in a NHS acute trust: the VOCAL mixed-methods study. NIHR Journals Library; 2018. doi: 10.3310/hsdr06210.
137. Braithwaite J, Wears R, Hollnagel E. Resilient health care: turning patient safety on its head. International Journal for Quality in Health Care. 2015; 27(5): 418–420.
138. Hollnagel E, Wears R, Braithwaite J. From Safety-I to Safety-II: A White Paper. University of Southern Denmark, University of Florida, USA, and Macquarie University, Australia; 2015.
139. The Health Foundation. A learning health system approach to avoiding acute occupancy crises [webpage]. The Health Foundation (www.health.org.uk/improvement-projects/a-learning-health-system-approach-to-avoiding-acute-occupancy-crises).
140. Fogel AL, Kvedar JC. Artificial intelligence powers digital medicine. npj Digital Medicine. 2018; 1:5.
141. NHS and Health Education England. Digital literacy: Towards a definition. NHS and Health Education England; 2016.
142. Lloyds Bank. UK Consumer Digital Index 2020. Lloyds Bank; 2020 (www.lloydsbank.com/assets/media/pdfs/banking_with_us/whats-happening/lb-consumer-digital-index-2020-report.pdf).
143. Davies R. The Topol Review – Preparing the healthcare workforce to deliver the digital future [webpage]. Good Things Foundation; 2019 (www.goodthingsfoundation.org/news-and-blogs/blog/topol-review-preparing-healthcare-workforce-deliver-digital-future).
144. The Health Foundation. Data-driven systems and health inequalities: COVID-19 and beyond [webpage]. The Health Foundation (www.health.org.uk/funding-and-partnerships/our-partnerships/data-driven-systems-and-health-inequalities).
145. Bosley S, Dale J. Healthcare assistants in general practice: practical and conceptual issues of skill-mix change. British Journal of General Practice. 2008; 58(547): 118–24.
146. The Health Foundation. Personalised medicine analytics dashboard for triaging patients in emergency departments [webpage]. The Health Foundation (www.health.org.uk/improvement-projects/personalised-medicine-analytics-dashboard-for-triaging-patients-in-emergency-departments).
147. Presentation at X-Gov Data Science Forum, 1 May 2020.
148. Spilsbury K, Adamson J, Atkin K, Bartlett C, Bloor K, Borglin G, Carr-Hill R, McCaughan D, McKenna H, Stuttard L, Wakefield A. Evaluation of the Development and Impact of Assistant Practitioners Supporting the Work of Ward-based Registered Nurses in Acute NHS (Hospital) Trusts in England. National Institute for Health Research; 2011.
149. Imison C, Castle-Clarke S, Watson R. Reshaping the workforce to deliver the care patients need. Nuffield Trust; 2016.
150. Ross J, Kettles A. Mental health nurse independent prescribing: what are nurse prescribers’ views of the barriers to implementation? J Psychiatr Ment Health Nurs. 2012; 19(10): 916–932.
151. Cabitza F, Rasoini R, Gensini GF. Unintended consequences of machine learning in medicine. JAMA. 2017; 318(6): 517–518.
152. Hoff T. Deskilling and adaptation among primary care physicians using two work innovations. Health Care Management Review. 2011; 36(4): 338–348. doi: 10.1097/HMR.0b013e31821826a1.
153. Buston, O. On the Edge [webpage]. Future Advocacy; 2018 (http://futureadvocacy.com/on-the-edge/).
154. Wears R, Cook R and Perry S. Automation, interaction, complexity, and failure: A case study. Reliability Engineering & System Safety. 2006; 91: 1494–1501.
155. Coiera E. How to ensure safety for medical artificial intelligence. AI Med; 1 August 2018 (https://ai-med.io/ai-med-news/safety-medical-artificial-intelligence/).
156. Tsai TL, Fridsma DB, Gatti G. Computer decision support as a source of interpretation error. J Am Med Inform Assoc. 2003; 10(5): 478–83.
157. Parasuraman, R. Manzey, DH. Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors. 2010; 52(3): 381–410. doi: 10.1177/0018720810376055.
158. Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A’Court C, Hinder S, Fahy N, Procter R, Shaw S. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res. 2017; 19(11): 367.
159. The Health Foundation, The King’s Fund and the Nuffield Trust. The health care workforce in England: Make or break? The Health Foundation, The King’s Fund and the Nuffield Trust; 2018 (www.health.org.uk/publications/the-health-care-workforce-in-england).
160. Steventon A, Deeny S, Keith J, Wolters A. Artificial intelligence in the NHS: getting the priorities right. The Health Foundation; 2019 (www.health.org.uk/news-and-comment/blogs/artificial-intelligence-in-the-nhs-getting-the-priorities-right).
161. Autor, D. Will Automation Take All of Our Jobs? TEDxCambridge; September 2016 (www.ted.com/talks/david_autor_will_automation_take_away_all_our_jobs).
162. Hammond J, Gravenhorst K, Funnell E, Beatty S, Hibbert D, Lamb J, Burroughs H, Kovandži M, Gabbay M, Dowrick C, Gask L, Waheed W, Chew-Graham CA. Slaying the dragon myth: an ethnographic study of receptionists in UK general practice. British Journal of General Practice. 2013; 63(608): 177–184.
163. Scottish Government. The Scottish Approach to Service Design: How to design services for and with users. Scottish Government; 2019 (www.gov.scot/publications/the-scottish-approach-to-service-design/).
164. Wears, R. Health Information Technology and Victory. Annals of Emergency Medicine. 2014; 65(2).
165. Landman AB, Redden L, Neri P, Poole S, Horsky J, Raja AS, Pozner CN, Schiff G, Poon EG. Using a medical simulation center as an electronic health record usability laboratory. J Am Med Inform Assoc. 2014; 21: 558–563.
166. Department of Health and Social Care. Guide to good practice for digital and data-driven health technologies. Department of Health and Social Care; January 2021 (www.gov.uk/government/publications/code-of-conduct-for-data-driven-health-and-care-technology/initial-code-of-conduct-for-data-driven-health-and-care-technology).
167. National Institute for Care Excellence. Evidence Standards Framework for Digital Health Technologies. National Institute for Care Excellence; March 2019 (www.nice.org.uk/Media/Default/About/what-we-do/our-programmes/evidence-standards-framework/digital-evidence-standards-framework.pdf).
168. The Health Foundation. Quality improvement made simple. The Health Foundation; 2013 (www.health.org.uk/publications/quality-improvement-made-simple).
169. The Health Foundation. Using communications approaches to spread improvement. The Health Foundation; 2013 (www.health.org.uk/publications/using-communications-approaches-to-spread-improvement).
170. Bardsley M, Steventon A, Fothergill G. Untapped potential: Investing in health and care data analytics. The Health Foundation; 2019 (www.health.org.uk/publications/reports/untapped-potential-investing-in-health-and-care-data-analytics).
171. SPIRIT-AI & CONSORT-AI: Improving The Quality Of Evidence For AI Health Interventions. [webpage]. 2020. (https://www.clinical-trials.ai/).
172. Susskind D, Susskind R. The Future of the Professions: How Technology Will Transform the Work of Human Experts. Oxford University Press; 2015.
173. Clarke G, Keith J, Steventon A. GP at Hand model will not work for everyone. HSJ; 5 June 2019 (www.hsj.co.uk/technology-and-innovation/gp-at-hand-model-will-not-work-for-everyone/7025226.article).