Sense about Science

Measures to tackle the Covid-19 pandemic brought unprecedented numbers of people into contact with the evidence behind government policy. Their demand for the background to government’s decisions – the justification for choices, the data that informed them and the statistics that showed what effect they had – produced a more extensive, diverse and sustained interface between society and government on policy evidence than ever before.  

As government rapidly developed policies to respond to the pandemic – 157 policies in the first few months of 2020 and similar rates with each roadmap – the public have found themselves grappling with the outputs of models and estimating risks. Over 40% of people viewed government websites to understand rules and evidence, while others sought that information from third parties. Researchers, actuaries and journalists who sought to understand the effects of Covid-19 policies found themselves mediating emerging government data to thousands of new followers.  

But the introduction of policies to manage Covid-19 did not just create a general public demand for policy evidence. The heads of organisations, from small businesses and community groups to large local authorities and National Health Service (NHS) trusts, had to assess and reassess which measures offered protection in their settings and communities, and what services they should run as each new phase of the pandemic and government controls unfolded. Schools had to explain closures. Health visitors had to insist on the importance of tackling non-Covid risks. Food distributors were torn between public service and employee protection. They too both sought and communicated what policy evidence was available.  

Fast policy innovation created a demand from experts and specialists too, inside and outside government, for explanations of what was known, where the gaps were and what needed to be done to fill them.  

How did government respond? Amid such huge public interest, politicians and government bodies in all four nations of the UK were faced with their own escalating need for policy evidence. They had to prioritise the questions, seek and manage the answers. These came with competing hypotheses, noisy data and wide-ranging outputs from epidemiological models. They also had to manage uncertainty and change while making rules and keeping advice clear. How does a government led by the science avoid defending the position that it has already picked and even enforced in law? 

People in all domains called for a better conversation about the evidence behind policy. Sense about Science, with our history of pursuing the public interest in policy evidence, has convened this scoping inquiry to capture the insights about how that conversation could be improved.  

While the pandemic is unique, the insights are far reaching. There is, for the first time, an opportunity to investigate at scale how government and society engage with policy evidence. It is also an opportunity to improve how government uses and shares data in the future. Many of the issues raised by the pandemic in this regard are raised by other situations where policy draws on complex and changing evidence about risks, and where interventions tackling one issue inevitably have repercussions across multiple other areas of society. We see them ahead in obesity reduction, emissions targets and flood prevention, and as far back as the outbreak of foot and mouth disease and the Clapham train crash. 

It is therefore important that the account of the pandemic acknowledges the successful innovations government made in providing the evidential reasoning for policy, as well as the frustrations. In addition, we all need to know what measures would make transparency and clarity about the evidence easier for government to achieve.  

We have published an open letter to Liz Truss and Rishi Sunak calling for a commitment to greater transparency and accountability in the decisions government makes.  

The letter calls on the Conservative Party leadership candidates to commit to standards of openness in the government’s use of scenarios and models – a problem highlighted by the Office for Statistics Regulation during the pandemic. It is pressing, not just in pandemic planning but in economic forecasts, climate impacts and energy policies. The next Prime Minister must ensure the public is entitled to the same transparency about inputs, assumptions and uncertainty that applies to other policy statistics.  

Summary

The What Counts? Scoping Inquiry was set up to explore what society needed to know and how well the UK government was able to respond. It identifies the significant questions that arise from the pandemic about the basis for government’s decisions and whether people across society could understand and apply it. The scale and extent of the interaction that occurred between society and government about policy evidence provided a unique opportunity for government to look at how it shares policy evidence and reasoning, and what will best serve society in the future – crisis or no crisis.  

This scoping inquiry set out to describe the challenges in more detail, and to find out what mattered to the many thousands of people who had to implement policy and draw on government reasoning to make decisions in their own settings. It took place between September 2021 and April 2022 and includes:  

• A nationwide survey in Autumn 2021, in partnership with NatCen Social Research, to find out how the UK public interacted with Covid-19 policy evidence. A report is in Appendix A, discussed in Section 1. 
• Interviews with, and submissions from, people providing facilities and services around the UK, who were faced with decisions about risks and trade-offs in a range of settings and communities. These are discussed in Section 1.  
• Interviews with and submissions from researchers and specialists about how evidence was sought and provided. These are discussed in Section 2. 
• An overview of the transparency of evidence, including a review of policies in the first two waves of the pandemic in 2020 (going into lockdown and emerging from lockdown) undertaken with colleagues at the Institute for Government. A report is in Appendix B, discussed in Section 2. 
• Discussions with government advisers, officials and politicians about the development of the evidence base and the challenges of communicating it throughout the crisis.  

The inquiry focused on the interface with policy evidence of the UK government. There were no appreciable differences from people surveyed in the devolved administrations. There were small differences in how the governments presented policy aims and evidence, and a larger inquiry could look at whether the effects of these are worth exploring.

Three fundamental questions have arisen from this inquiry: 

• Does government seek to enable people with the information it provides?  
• Does government have the tools to generate, assimilate and communicate the evidence in all areas of policy?  
• What are the value frameworks for weighing up competing risks and making trade-offs between them?  

This report is the start of a conversation about these, shaped by people’s experiences in practice. 

Findings

1. The Covid-19 pandemic led to unprecedented interest in policy evidence and the government had an ongoing window of popular engagement with its pandemic information: 69% got information about the pandemic from government briefings, though just 30% considered these the most useful source of Covid-19 information provided by the government.
2. People sought information on a wide range of issues. Most people felt the government did not have enough evidence about the effect of the pandemic and measures to combat it, with only 6% of respondents not identifying any area where there was a deficit. 
3. Society is made up of hundreds of thousands of intermediaries of policy information, from rural headteachers and health visitors, to store managers and sports coaches, who sought evidence and reasoning to implement measures and make trade-offs with other risks they managed – but government on the whole did not appear to operate in a way that reflected this, opting instead for simple messaging.  
4. Rapid policy innovation during the crisis circumvented the usual processes for developing and communicating policy. In the first months of the pandemic, 60% of policies (96) were set out in a press release rather than a policy document. Evaluation of a sample of policies against the Evidence Transparency Framework found that around 90% provided no clear link to available supporting evidence for relevant parts of the policy. 
5. Government did not to set out the purpose of policies with reference to an ethical framework nor according to established cost-benefit principles of health economics, nor according to accepted definitions of health, which includes social wellbeing and mental health. No treasury models were published and there appeared to be squeamishness about discussing QALYs, even though they are a fundamental tool in government. 

We have characterised the government’s communication of evidence centrally as largely authoritarian, with its emphasis on simple messages and rules.  

6. Simple instructions were intended to provide clarity in public health messages, but in practice left people unclear in many significant sectors. 
7. The approach appeared successful initially, lending itself to slogans such as ‘Hands, Face, Space’, ‘Stay Home, Save Lives, Protect the NHS’ and ‘the Rule of Six’, but it presented government and society with difficulties when rules and guidance had to change to reflect new evidence or shifting priorities. 
8. People understood a lack of information but struggled with sudden changes of direction that lacked explanation. The government passed over the opportunity to inform people more fully about what was knowable and what was not and to prepare them for what might change. The focus on rules meant when rules were no longer workable in practice or appeared ineffective – or they were broken by prominent people – authority was lost.  
9. People with responsibilities in all sectors found government’s sharing of the evidence it was using inadequate for making trade-offs with other risks they manage such as protecting children, cancer care or avoiding redundancies.  
10. In Autumn 2021, around half of people thought the government was not at all aware or only a little aware of how the pandemic and measures to combat it affected households like theirs and people in their line of work.  
11. During periods of opening up, many organisations did not feel that they had the evidence needed about the degree of risk in their sector or community, which differed from the average. People wanted a sense of scale about Covid-19 risks and the respective benefits of measures. 
12. People were unsure about which measures to stop emphasising because they could not see whether the evidence had moved on or another priority had intervened.  
13. People running services felt that government’s ‘everyone is at risk’ message, designed to achieve compliance, made it harder to focus measures on those who were far more at risk, and harder to protect services for children. 
14. Despite the aims of simple messages, polling by Ipsos MORI found people misunderstood the risks and found rules hard to follow as time went on.  
15. As well as looking for evidence, people sought policy goals that could help them prioritise. It was not clear what society’s goal was supposed to be: even in narrow terms of protecting the NHS, reducing R and lowering cases – how did people know when they had done enough? The goal clearly was not to stop Covid at any cost, but what was the tolerable cost? 
16. Models have been central to public discussion about Covid-19 measures. Simple directive communication about policy made it hard to communicate the purpose and limits of models, and simple graphs got overstated, leading to a backlash against scientific advice in some parts of society.  
17. The conversation between politicians and evidence providers also suffered from the lack of discussion about policy objectives. Modellers did not know what social needs government sought to optimise. 
18. Contrary to the apparent aim of message control, when government did not give justifications for decisions, others were drawn into the vacuum. Scientific advisers were pushed to speak beyond their expertise and some people spoke with none. 
19. The government expressed growing concern with misinformation and established a unit to deal with it in the Cabinet Office. It was puzzling that government believed its role to be chasing discussions on the margins of social comment, and yet it was doing too little to respond to the large numbers of people actively scouring government websites and sources for a better understanding of decisions and evidence. 

We have identified more authoritative efforts in government to provide and discuss the evidence base and how decisions might change. 

20. Although many things were and remain unknown about the pandemic, the working knowledge that government had presented the opportunity to talk about gaps and difficulty in knowledge from the start, which would have facilitated more nuanced conversations about rule changes and new data later. 
21. Community and sector leaders made a lot of use of those initiatives and efforts to provide evidence for them to use in their own situations. The medical and scientific advisers and government statisticians were notable in doing this. 
22. Understanding the goal of a policy made it easier for intermediaries and people with responsibilities to implement in a real-world setting, and to identify other opportunities to achieve the goal. 
23. The speed and severity of Covid-19’s arrival meant that arrangements to gather evidence had to be made alongside early decisions. The UK government was world-leading in its creation and sharing of data sources to track cases, hospitalisations and deaths.  
24. The public interfaces and data gathering and sharing by Public Health England (PHE; now UK Health Security Agency) and the Office for National Statistics (ONS) continued to develop throughout the pandemic in response to the needs of their (growing) groups of users. Engagement with the PHE data dashboard and ONS websites was surprisingly high (35% and 15% of respondents). 
25. SAGE and its sub committees were successful in meeting the demands of sudden public attention, despite starting from a position of not even having a public facing website. A heavy burden was placed on scientific advice to explain policy; very little was carried by policy advisers or economic advisers in government. 
26. The crisis has exposed the poor organisation of government websites and social media. Officials found the gov.uk website a poor tool for organising material about the pandemic. 
27. In respect of the extensive use of modelling and scenario development during the pandemic, the Office for Statistics Regulation said in October 2021 that “principles of the Code of Practice and our expectations on transparency should be followed for any analysis published by governments”. 
28. Programmes to investigate the virus and treatments were set up effectively in the UK, and communicated by universities, journals, government and journalists. The government funded and collaborated in research programmes and built out from existing strengths in genetic sequencing and vaccine research. This may be why people raised few concerns about the biomedical evidence behind policy and were able to raise important questions about the absence of studies, such as Covid-19 in Black and Asian people and the omission of children and pregnant women from vaccine trials. 
29. While people were frustrated by the introduction of vaccine mandates and passports without evidence about what they would achieve in venues, the government communicated the role out of vaccination programme clearly and with reasons people understood.

Recommendations

30. Investigation of whether an authoritarian approach with simple messaging has a limited place, such as in emergencies of shorter duration or simpler implications. Confident knowledge of this might help government to keep simple public health messaging as a tool in a crisis rather than risk subverting society’s wider information needs. 
31. The government should consider setting up a socially responsive trials unit, which could be scaled up to major policy actions in the future, building on the What Works initiatives. It would also be a conduit to report the questions that are arising in society. 
32. Politicians and policy advisors should consider whether, in emphasising simple messaging, it is leaving others to provide the analysis on which people act; it should seek to understand the role played by intermediaries of government data, such as the actuaries on Twitter, and to learn from individuals within government who took on the role of guiding people through evidence in social media discussions. 
33. Government should review whether modelling has been under-deployed with respect to exploring how to meet competing goals, and how future pan-departmental policy issues and crisis responses might use modelling to optimise interventions to do the most good for the least harm. 
34. The Cabinet Office and the Treasury should implement a plan to ensure full publication of models used in policy making, including their assumptions, code and scenarios. As well as being a democratic principle of people having access to the same information, it may have practical benefits. Process assumptions are often hidden but could give more people intuition about why the model says what it does and what aspects of life it’s relevant for. 
35. Government should reflect on the large burden of communication put on its emergency committee. Because inputs to the scientific advice were transparent, whereas economic advice and policy advice were not, there was too much focus on scientific advice to provide the rationale for policy. 
36. Government should also consider the associated risk of its emergency advice body being concerned with what the public and media think.
37. In addition, attention is needed to the question of whether commandeering the strongest independent modelling centres for long periods creates risks, and whether crisis management requires some modellers independent of the constraints given by policy makers. 
38. Government should investigate whether rapid sharing of useful evidence with stakeholder groups may have been hampered by reducing the leadership of policies by dedicated departments. 
39. Government should consider how it might set a transparency of evidence standard.  
40. Government should use the experience of the mass engagement of the crisis to restructure gov.uk in a way that is responsive and accessible to people looking for policy evidence and rationale.  
41. Both government and the public need to be better equipped to discuss difficult trade-offs and uncertainty. People need to know what counts – what value framework is being used to weigh up competing risks and benefits and make trade-offs between them. 
42. If government sees its role as enabling society, both in the implementation of policy and in making well-reasoned judgements in the many and varied settings, then it should explore what this approach means in practice. As a start, the findings of this inquiry are that an enabling approach would be characterised by goals of competence, transparency and reflexiveness.

Our scoping has found that, as a start, an enabling approach by the government would be characterised by goals of competence, transparency and reflexiveness.