Critical Appraisal of Quantitative Research

2014

Background Recently there has been a significant increase in the number of systematic reviews addressing questions of prevalence. Key features of a systematic review include the creation of an a priori protocol, clear inclusion criteria, a structured and systematic search process, critical appraisal of studies, and a formal process of data extraction followed by methods to synthesize, or combine, this data. Currently there exists no standard method for conducting critical appraisal of studies in systematic reviews of prevalence data. Methods A working group was created to assess current critical appraisal tools for studies reporting prevalence data and develop a new tool for these studies in systematic reviews of prevalence. Following the development of this tool it was piloted amongst an experienced group of sixteen healthcare researchers. Results The results of the pilot found that this tool was a valid approach to assessing the methodological quality of studies reporting prevalence data to be included in systematic reviews. Participants found the tool acceptable and easy to use. Some comments were provided which helped refine the criteria. Conclusion The results of this pilot study found that this tool was well-accepted by users and further refinements have been made to the tool based on their feedback. We now put forward this tool for use by authors conducting prevalence systematic reviews.

The term critical appraisal of the literature, as used in the context of evidence-based medicine (EBM), refers to the application of predefined rules of evidence to a study to assess its methodological quality and the clinical usefulness of its results. Critical appraisal represents the most technical step in the process of EBM and can be quite demanding for the practitioner. The aim of this paper is to provide the reader with the theoretical skills necessary to understand the principles behind critical appraisal of the literature. These include: (a) the description of the main types of study design used in epidemiological research, (b) the basic statistical procedures used in data analysis, (c) the principles of causal inference and (d) the description of the types of health outcome and measures of effect. These issues are discussed in the present paper and illustrated with several examples from the relevant literature.

Systematic reviews play a crucial role in evidence-based practices as they consolidate research findings to inform decision-making. However, it is essential to assess the quality of systematic reviews to prevent biased or inaccurate conclusions. This paper underscores the importance of adhering to recognized guidelines, such as the PRISMA statement and Cochrane Handbook. These recommendations advocate for systematic approaches and emphasize the documentation of critical components, including the search strategy and study selection. A thorough evaluation of methodologies, research quality, and overall evidence strength is essential during the appraisal process. Identifying potential sources of bias and review limitations, such as selective reporting or trial heterogeneity, is facilitated by tools like the Cochrane Risk of Bias and the AMSTAR 2 checklist. The assessment of included studies emphasizes formulating clear research questions and employing appropriate search strategies to construct robust reviews. Relevance and bias reduction are ensured through meticulous selection of inclusion and exclusion criteria. Accurate data synthesis, including appropriate data extraction and analysis, is necessary for drawing reliable conclusions. Meta-analysis, a statistical method for aggregating trial findings, improves the precision of treatment impact estimates. Systematic reviews should consider crucial factors such as addressing biases, disclosing conflicts of interest, and acknowledging review and methodological limitations. This paper aims to enhance the reliability of systematic reviews, ultimately improving decision-making in healthcare, public policy, and other domains. It provides academics, practitioners, and policymakers with a comprehensive understanding of the evaluation process, empowering them to make well-informed decisions based on robust data.

BMJ

AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both The number of published systematic reviews of studies of healthcare interventions has increased rapidly and these are used extensively for clinical and policy decisions. Systematic reviews are subject to a range of biases and increasingly include non-randomised studies of interventions. It is important that users can distinguish high quality reviews. Many instruments have been designed to evaluate different aspects of reviews, but there are few comprehensive critical appraisal instruments. AMSTAR was developed to evaluate systematic reviews of randomised trials. In this paper, we report on the updating of AMSTAR and its adaptation to enable more detailed assessment of systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. With moves to base more decisions on real world observational evidence we believe that AMSTAR 2 will assist decision makers in the identification of high quality systematic reviews, including those based on non-randomised studies of healthcare interventions.

Handbook of Theory and Methods in Applied Health Research, 2020

International Journal of Nursing Studies

Systematic literature reviews identify, select, appraise, and synthesize relevant literature on a particular topic. Typically, these reviews examine primary studies based on similar methods, e.g., experimental trials. In contrast, interest in a new form of review, known as mixed studies review (MSR), which includes qualitative, quantitative, and mixed methods studies, is growing. In MSRs, reviewers appraise studies that use different methods allowing them to obtain in-depth answers to complex research questions. However, appraising the quality of studies with different methods remains challenging. To facilitate systematic MSRs, a pilot Mixed Methods Appraisal Tool (MMAT) has been developed at McGill University (a checklist and a tutorial), which can be used to concurrently appraise the methodological quality of qualitative, quantitative, and mixed methods studies.The purpose of the present study is to test the reliability and efficiency of a pilot version of the MMAT.The Center for Participatory Research at McGill conducted a systematic MSR on the benefits of Participatory Research (PR). Thirty-two PR evaluation studies were appraised by two independent reviewers using the pilot MMAT. Among these, 11 (34%) involved nurses as researchers or research partners. Appraisal time was measured to assess efficiency. Inter-rater reliability was assessed by calculating a kappa statistic based on dichotomized responses for each criterion. An appraisal score was determined for each study, which allowed the calculation of an overall intra-class correlation.On average, it took 14 min to appraise a study (excluding the initial reading of articles). Agreement between reviewers was moderate to perfect with regards to MMAT criteria, and substantial with respect to the overall quality score of appraised studies.The MMAT is unique, thus the reliability of the pilot MMAT is promising, and encourages further development.

Cardiopulmonary Physical Therapy Journal, 2001

Critical review of the literature in order to provide high quality, evidence-based care to our clients can be facilitated by reading systematic reviews. Part II: Grading the Evidence-will describe the various aspects of reviewing the quality of research studies, combining study results in systematic reviews, and assessing the quality of review articles. Clinical trials can be graded by various schemes outlined in the literature that examines the research design, processes of randomization and blinding, and reliability and validity of outcomes. Systematic reviews, which use scientific strategies to assemble, appraise and integrate outcomes from different studies, may be a more objective way to review the literature than narrative reviews. A meta-analysis is a systematic review that uses statistical methods to combine data from various studies. The merits and shortcomings of meta-analyses and grading studies will be described. Lastly, issues related to data validity, its presentation, and transformation will be outlined. In conclusion, clinicians should avoid scanning research papers and accepting the conclusions at face value. By applying some of the basic principles outlined in this paper, the clinician can scrutinize the scientific rigor of the methods outlined in research papers including review papers and thus, assess the merit of the results and conclusions outlined.

Journal of Evaluation in Clinical Practice, 2012

The Cochrane Collaboration is strongly encouraging the use of a newly developed tool, the Cochrane Collaboration Risk of Bias Tool (CCRBT), for all review groups. However, the psychometric properties of this tool to date have yet to be described. Thus, the objective of this study was to add information about psychometric properties of the CCRBT including inter-rater reliability and concurrent validity, in comparison with the Effective Public Health Practice Project Quality Assessment Tool (EPHPP). Methods Both tools were used to assess the methodological quality of 20 randomized controlled trials included in our systematic review of the effectiveness of knowledge translation interventions to improve the management of cancer pain. Each study assessment was completed independently by two reviewers using each tool. We analysed the inter-rater reliability of each tool's individual domains, as well as final grade assigned to each study.

Academic radiology, 2014

Recent efforts have been made to standardize the critical appraisal of clinical health care research. In this article, critical appraisal of diagnostic test accuracy studies, screening studies, therapeutic studies, systematic reviews and meta-analyses, cost-effectiveness studies, recommendations and/or guidelines, and medical education studies is discussed as are the available instruments to appraise the literature. By having standard appraisal instruments, these studies can be appraised more easily for completeness, bias, and applicability for implementation. Appraisal requires a different set of instruments, each designed for the individual type of research. We also hope that this article can be used in academic programs to educate the faculty and trainees of the available resources to improve critical appraisal of health research.

European Journal of Epidemiology

To inform evidence-based practice in health care, guidelines and policies require accurate identification, collation, and integration of all available evidence in a comprehensive, meaningful, and time-efficient manner. Approaches to evidence synthesis such as carefully conducted systematic reviews and meta-analyses are essential tools to summarize specific topics. Unfortunately, not all systematic reviews are truly systematic, and their quality can vary substantially. Since well-conducted evidence synthesis typically involves a complex set of steps, we believe formulating a cohesive, step-by-step guide on how to conduct a systemic review and meta-analysis is essential. While most of the guidelines on systematic reviews focus on how to report or appraise systematic reviews, they lack guidance on how to synthesize evidence efficiently. To facilitate the design and development of evidence syntheses, we provide a clear and concise, 24-step guide on how to perform a systematic review and meta-analysis of observational studies and clinical trials. We describe each step, illustrate it with concrete examples, and provide relevant references for further guidance. The 24-step guide (1) simplifies the methodology of conducting a systematic review, (2) provides healthcare professionals and researchers with methodologically sound tools for conducting systematic reviews and meta-analyses, and (3) it can enhance the quality of existing evidence synthesis efforts. This guide will help its readers to better understand the complexity of the process, appraise the quality of published systematic reviews, and better comprehend (and use) evidence from medical literature.