Evidence Synthesis Hackathon Summary Report 2018

Environmental Evidence, 2020

Evidence synthesis is a vital part of evidence-informed decision-making, but high growth in the volume of research evidence over recent decades has made efficient evidence synthesis increasingly challenging. As the appreciation and need for timely and rigorous evidence synthesis continue to grow, so too will the need for tools and frameworks to conduct reviews of expanding evidence bases in an efficient and time-sensitive manner. Efforts to future-proof evidence synthesis through the development of new evidence synthesis technology (ESTech) have so far been isolated across interested individuals or groups, with no concerted effort to collaborate or build communities of practice in technology production. We established the evidence synthesis Hackathon to stimulate collaboration and the production of Free and Open Source Software and frameworks to support evidence synthesis. Here, we introduce a special series of papers on ESTech, and invite the readers of environmental evidence to submit manuscripts introducing and validating novel tools and frameworks. We hope this collection will help to consolidate ESTech development efforts and we encourage readers to join the ESTech revolution. In order to future-proof evidence synthesis against the evidence avalanche, we must support community enthusiasm for ESTech, reduce redundancy in tool design, collaborate and share capacity in tool production, and reduce inequalities in software accessibility.

Nature Ecology & Evolution, 2020

Synthesizing evidence is an essential part of scientific progress, but it is often done in a slow and uncoordinated manner, sometimes producing misleading conclusions. Here, we propose the idea of an 'open synthesis community' to resolve this pressing issue.

Evidence-based environmental management is being hindered by difficulties in locating, interpreting and synthesising relevant information among vast scientific outputs. But software developments that allow enhanced collation and sharing of data will help.

Achieving evidence-based environmental management requires that decision-makers have access to evidence that can help identify the most effective interventions for their management context. Evidence synthesis supports evidence-based decision-making because it collates, filters and makes sense of a sometimes large and often conflicting evidence-base, potentially yielding new insights. There are many approaches to evidence synthesis. They each have different strengths and weaknesses, making them suited to different purposes, questions and contexts, given particular constraints. To make sense of the wide array of approaches, we outline the important considerations when selecting the most appropriate method for a particular decision context. These include the purpose for the synthesis, the required outcomes, and the multiple constraints within which decision-makers must operate. We then critically assess a spectrum of approaches to evidence synthesis commonly used within environmental management, detailing the characteristics of each that can be used to determine when it is a suitable method. To guide this selection process we provide a decision tree for those commissioning (e.g., decision-makers or stakeholders) or conducting (e.g., scientists) evidence synthesis, which can be used to identify an appropriate method. The decision tree classifies evidence synthesis methods according to whether their purpose is to test or generate hypotheses, the level of resources they require, the level of certainty in the outputs, and the type and scope of the question being addressed. This tool is a major advance because it helps select an appropriate synthesis method based on the multiple constraints that impact the decision. We conclude that there is an approach to evidence synthesis that will suit all management contexts, but that selecting the right approach requires careful consideration of what is fit for purpose.

Environmental Evidence, 2021

One of the most important steps in the process of conducting a systematic review or map is data extraction and the production of a database of coding, metadata and study data. There are many ways to structure these data, but to date, no guidelines or standards have been produced for the evidence synthesis community to support their production. Furthermore, there is little adoption of easily machine-readable, readily reusable and adaptable databases: these databases would be easier to translate into different formats by review authors, for example for tabulation, visualisa-tion and analysis, and also by readers of the review/map. As a result, it is common for systematic review and map authors to produce bespoke, complex data structures that, although typically provided digitally, require considerable efforts to understand, verify and reuse. Here, we report on an analysis of systematic reviews and maps published by the Collaboration for Environmental Evidence, and discuss major issues that hamper machine readability and data reuse or verification. We highlight different justifications for the alternative data formats found: condensed databases; long databases; and wide databases. We describe these challenges in the context of data science principles that can support curation and publication of machine-readable, Open Data. We then go on to make recommendations to review and map authors on how to plan and structure their data, and we provide a suite of novel R-based functions to support efficient and reliable translation of databases between formats that are useful for presentation (condensed, human readable tables), filtering and visualisation (wide databases), and analysis (long databases). We hope that our recommendations for adoption of standard practices in database formatting, and the tools necessary to rapidly move between formats will provide a step-change in transparency and replicability of Open Data in evidence synthesis.

Environmental Evidence, 2023

In civil society we expect that policy and management decisions will be made using the best available evidence. Yet, it is widely known that there are many barriers that limit the extent to which that occurs. One way to overcome these barriers is via robust, comprehensive, transparent and repeatable evidence syntheses (such as systematic reviews) that attempt to minimize various forms of bias to present a summary of existing knowledge for decision-making purposes. Relative to other disciplines (e.g., health care, education), such evidence-based decision-making remains relatively nascent for environment management despite major threats to humanity, such as the climate, pollution and biodiversity crises demonstrating that human well-being is inextricably linked to the biophysical environment. Fortunately, there are a growing number of environmental evidence syntheses being produced that can be used by decision makers. It is therefore an opportune time to reflect on the science and practice of evidence-based decision-making in environment management to understand the extent to which evidence syntheses are embraced and applied in practice. Here we outline a number of key questions related to the use of environmental evidence that need to be explored in an effort to enhance evidence-based decision-making. There is an urgent need for research involving methods from social science, behavioural sciences, and public policy to understand the basis for patterns and trends in environmental evidence use (or misuse or ignorance). There is also a need for those who commission and produce evidence syntheses, as well as the end users of these syntheses to reflect on their experiences and share them with the broader evidence-based practice community to identify needs and opportunities for advancing the entire process of evidence-based practice. It is our hope that the ideas shared here will serve as a roadmap for additional scholarship that will collectively enhance evidence-based decision-making and ultimately benefit the environment and humanity.

Campbell Systematic Reviews, 2021

Environmental Evidence, 2018

The Open Science movement can be broadly summarised as aiming to promote integrity, repeatability and transparency across all aspects of research, from data collection to publication. Systematic reviews and systematic maps aim to provide a reliable synthesis of the evidence on a particular topic, making use of methods that seek to maximise repeatability and comprehensives whilst minimising subjectivity and bias. The central tenet of repeatability is opera-tionalised by transparently reporting methodological activities in detail, such that all actions could be replicated and verified. To date, evidence synthesis has only partially embraced Open Science, typically striving for Open Methodology and Open Access, and occasionally providing sufficient information to be considered to have Open Data for some published reviews. Evidence synthesis communities needs to better embrace Open Science not only to balance knowledge access and increase efficiency, but also to increase reliability, trust and reuse of information collected and synthesised within a review: concepts fundamental to systematic reviews and maps. All aspects of Open Science should be embraced: Open Methodology, Open Data, Open Source and Open Access. In doing so, evidence synthesis can be made more equal, more efficient and more trustworthy. I provide concrete recommendations of how CEE and others can fully embrace Open Synthesis.

Conservation science and practice, 2021

Practitioners and policymakers working in environmental arenas make decisions that can have large impacts on ecosystems. Basing such decisions on highquality evidence about the effectiveness of different interventions can often maximize the success of policy and management. Accordingly, it is vital to understand how environmental professionals working at the science-policy interface view and use different types of evidence, including evidence syntheses that collate and summarize available knowledge on a specific topic to save time for decision-makers. We interviewed 84 senior environmental professionals in Canada working at the science-policy interface to explore their confidence in, and use of, evidence syntheses within their organizations. Interviewees value evidence syntheses because they increase confidence in decision-making, particularly for high-profile or risky decisions. Despite this enthusiasm, the apparent lack of available syntheses for many environmental issues means that use can be

Environmental Evidence, 2017

The first international Collaboration for Environmental Evidence (CEE) conference took place in August 2016 at the Swedish Museum of Natural History in Stockholm with nearly 100 participants from 14 countries. This conference reflected and contributed to the growth of a global network of people interested in the production and use of evidence syntheses in environmental management. The conference also provided an opportunity to identify emerging themes and reflect on those ideas and perspectives to help direct future activities of the CEE and the broader community. An increasingly engaged community of practice was evident but there is uneven distribution of experience, resources, capacity, and commitment to evidence synthesis in different sectors and regions. There is much opportunity to bring academics, practitioners, and other partners together which will help to further demonstrate impact of evidence synthesis activities and enhance relevance. As the discipline evolves there is growing interest in rapid evidence synthesis but the benefits and risks of that approach remain unclear. There was also a recognition that improvements in empirical science will enhance the likelihood that more studies can be fully exploited as part of evidence synthesis. There are opportunities for capacity building, engaging the next generation (e.g., students), and enhancing connections within and beyond the CEE community to advance evidence-based environmental management. It is our desire that this paper will serve as a template for future CEE activities (i.e., where to invest resources) but also as an invitation to those that were unable to attend to participate in CEE and the evidence-based environmental management movement in whichever ways resonate with them.