By 2019, researchers were publishing roughly 80 systematic reviews every single day — a more than 20-fold increase compared to the year 2000. With that flood of evidence syntheses came a new problem: how do you make sense of dozens of systematic reviews on the same broad topic when they use different methods, cover overlapping populations, and sometimes reach contradictory conclusions? The umbrella review was developed to solve exactly this challenge. If you have ever tried to reconcile conflicting systematic reviews or needed a high-level summary of all available evidence on a research question, understanding umbrella reviews is essential to your work.
This guide covers what an umbrella review is, how it differs from other review types, when you should use one, and the step-by-step process for conducting one from protocol to publication.
What is an umbrella review?
An umbrella review is a systematic synthesis of previously published systematic reviews and meta-analyses on a specific research topic. Rather than analyzing individual primary studies, it treats each systematic review as its unit of analysis, aggregating findings across multiple reviews to produce a comprehensive, high-level evidence summary.
Umbrella reviews are also known as overviews of reviews, reviews of reviews, meta-reviews, or syntheses of reviews. The methodology was popularized in epidemiology and biomedical research but has since expanded into psychology, education, public health, and other disciplines.
According to BMJ Medicine, umbrella reviews were developed to deal with the increasing number of systematic reviews and meta-analyses in biomedical literature. Their key output is a standardized assessment of all available evidence on a well-defined research topic — for example, evaluating treatment effects of multiple interventions for a particular condition, or comparing risk factors associated with a specific disease across all published meta-analyses.
Because umbrella reviews sit at the top of the evidence hierarchy — synthesizing evidence that has already been systematically synthesized — they are considered one of the highest levels of evidence currently available.
Umbrella review vs. systematic review: what is the difference?
The core difference is the unit of analysis. A systematic review collects and analyzes primary studies (clinical trials, observational studies, qualitative research). An umbrella review collects and analyzes systematic reviews and meta-analyses themselves.
Here is how they compare across key dimensions:
A systematic review asks: "What does the primary evidence say about X?" An umbrella review asks: "What do all the systematic reviews say about X, and how consistent, reliable, and complete is that body of synthesized evidence?"
How does an umbrella review differ from a scoping review?
A scoping review maps the breadth of available literature on a topic — identifying research gaps, key concepts, and the types of evidence that exist — without necessarily assessing quality. An umbrella review, by contrast, only includes systematic reviews, critically appraises them, and synthesizes their quantitative or qualitative findings. Scoping reviews are exploratory; umbrella reviews are summative and evaluative.
How does an umbrella review differ from a meta-analysis?
A meta-analysis statistically pools results from individual primary studies to produce a combined effect estimate. An umbrella review may report and compare the effect estimates from multiple meta-analyses, but it does not re-pool primary study data. Instead, it evaluates consistency, heterogeneity, and quality across the meta-analytic results that already exist.
When should you conduct an umbrella review?
An umbrella review is the right methodology when several conditions are met. You should consider one when:
Multiple systematic reviews already exist on your topic or closely related subtopics. If only one or two systematic reviews have been published, an umbrella review adds limited value — a standard systematic review or an update of existing reviews may be more appropriate.
Existing reviews reach different conclusions. When published systematic reviews on the same question report conflicting effect sizes, use different inclusion criteria, or cover overlapping but distinct study pools, an umbrella review provides a structured way to compare and reconcile these differences.
You need a broad evidence overview for clinical guidelines, policy decisions, or research priority-setting. Umbrella reviews are especially valuable for decision-makers who need to understand the full landscape of evidence without reading dozens of individual reviews.
The topic is controversial or impacted by bias. Controversial topics in health, education, or social science often generate numerous reviews of varying quality. An umbrella review can assess methodological rigor across all available syntheses and stratify evidence by quality.
You are writing a grant proposal or research agenda and need to demonstrate the current state of synthesized evidence on a broad area, including where gaps remain.
Umbrella reviews are not appropriate when too few systematic reviews exist on your topic, when you need to answer a narrow clinical question better served by a single focused systematic review, or when the available reviews are too heterogeneous in scope to meaningfully compare.
How to conduct an umbrella review: a step-by-step guide
The methodology for conducting an umbrella review closely mirrors that of a systematic review, with key differences in inclusion criteria, data extraction, and quality assessment. Here is the process broken down into actionable stages.
Step 1: Define the research question and justify the need
Before starting, confirm that an umbrella review is the appropriate methodology. Search PROSPERO, the Cochrane Database, and PubMed to verify that no recent umbrella review on your topic already exists. Your research question should be broad enough to encompass multiple systematic reviews but specific enough to produce meaningful, actionable conclusions.
Use the PICO framework (Population, Intervention, Comparator, Outcome) or a variation of it to structure your question. For example: "What is the current synthesized evidence on the effectiveness of mindfulness-based interventions (I) compared to standard care (C) for anxiety reduction (O) in university students (P) across published systematic reviews and meta-analyses?"
Step 2: Write and register the protocol
A pre-registered protocol increases transparency and reduces the risk of post-hoc modifications that could introduce bias. Register your protocol on PROSPERO (the International Prospective Register of Systematic Reviews) and follow the PRISMA-P 2015 guidelines for protocol reporting.
Your protocol should specify:
Eligibility criteria (which types of systematic reviews will be included)
Databases to be searched
Search strategy
Screening and selection process
Data extraction variables
Quality assessment tool (typically AMSTAR 2)
Method for synthesizing results
Step 3: Develop and execute the search strategy
Search at least three to four major databases — MEDLINE (PubMed), Embase, Cochrane Database of Systematic Reviews, and a discipline-specific database relevant to your field. Use search terms that combine your topic-specific keywords with review methodology filters (e.g., "systematic review," "meta-analysis," "evidence synthesis").
Unlike a standard systematic review where you might screen thousands of primary studies, umbrella review searches typically yield a more manageable number of results because you are filtering exclusively for review-level publications. However, comprehensive searching is still critical — missing a key systematic review can significantly affect your conclusions.
Supplement database searches with manual reference checking of included reviews, citation tracking, and grey literature searches where appropriate.
Step 4: Screen and select reviews
Apply your predefined eligibility criteria in a two-stage process: title and abstract screening followed by full-text assessment. At least two independent reviewers should screen records to minimize bias, with disagreements resolved by consensus or a third reviewer.
Tools like Rayyan and Covidence can streamline the screening process, especially for larger searches. Document the entire selection process and prepare a PRISMA flow diagram showing the number of records identified, screened, excluded (with reasons), and included.
Step 5: Extract data from included reviews
Create a standardized data extraction form that captures:
Bibliographic details — authors, year, journal, country
Review characteristics — type of review, number of primary studies included, total participants, databases searched, date range of included studies
Population, intervention, comparator, and outcome details
Key findings — effect sizes (odds ratios, risk ratios, mean differences), confidence intervals, statistical significance
Heterogeneity measures — I² statistics, prediction intervals
Quality assessment scores from the original review (if reported)
Funding sources and declared conflicts of interest
At least two reviewers should extract data independently and cross-check for accuracy. This is where the organizational complexity of an umbrella review becomes apparent — you are managing data from multiple reviews, each of which contains data from multiple primary studies.
Step 6: Assess the methodological quality of included reviews
Use a validated tool to critically appraise each included systematic review. The most widely used tool for this purpose is AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews), which evaluates 16 domains including protocol registration, comprehensive searching, duplicate screening, risk of bias assessment, and appropriate statistical methods.
The JBI (Joanna Briggs Institute) Critical Appraisal Checklist for Systematic Reviews is another commonly used option. Whichever tool you choose, report quality ratings transparently and consider stratifying your results by review quality — conclusions drawn from high-quality reviews carry more weight than those from reviews with critical methodological weaknesses.
Step 7: Synthesize results and assess overlap
Present your findings in a structured format. If the included reviews report comparable quantitative outcomes, you can summarize effect estimates in a table or forest plot. If reviews are too heterogeneous for quantitative comparison, a narrative synthesis organized by subtopic, population, or intervention type is appropriate.
A critical step unique to umbrella reviews is assessing overlap among the primary studies included in the different systematic reviews. Because multiple reviews on the same topic often include some of the same primary studies, there is a risk of over-counting evidence. Calculate and report the corrected covered area (CCA) or use a citation matrix to show the degree of overlap.
Step 8: Grade the evidence and report transparently
Stratify your evidence using a classification system. Several umbrella review-specific grading frameworks exist, including the system proposed by Fusar-Poli and Radua, which classifies evidence as convincing, highly suggestive, suggestive, weak, or non-significant based on statistical significance, sample size, heterogeneity, prediction intervals, and potential biases.
Report your umbrella review following PRISMA 2020 guidelines. While PRISMA was originally designed for systematic reviews, its principles of transparent, complete reporting apply equally to umbrella reviews. Include your search strategy, PRISMA flow diagram, characteristics of included reviews, quality assessment results, synthesis of findings, and an honest discussion of limitations.
Common challenges in umbrella reviews and how to overcome them
Managing the multi-layered evidence structure
The biggest organizational challenge in an umbrella review is managing layers of evidence — your review contains systematic reviews, each of which contains primary studies, each of which contains data points. Keeping track of which primary studies appear in which reviews, which outcomes are reported where, and how review-level and study-level data connect requires careful, structured documentation.
Research teams often struggle with scattered files — extraction spreadsheets in one location, PDFs in another, quality assessment forms in a shared drive, and protocol drafts in email threads. A research project management platform like ScholarDock can help by keeping all these materials — source PDFs, extraction tables, quality assessments, and team task assignments — connected in a single organized workspace where every document is linked to the project it belongs to.
Dealing with overlapping primary studies
When three systematic reviews on the same topic each include many of the same randomized controlled trials, the overlap inflates the apparent volume of evidence. Calculating the CCA and building a citation matrix requires meticulous cross-referencing of primary study lists across all included reviews. This is tedious but essential — failing to account for overlap is one of the most common methodological criticisms of umbrella reviews.
Handling heterogeneous reviews
Not all systematic reviews are created equal. Some include only RCTs; others mix observational and experimental designs. Some cover adult populations; others include children. Your umbrella review must transparently document these differences and, where possible, conduct subgroup analyses or sensitivity analyses to explore how review-level differences affect the overall conclusions.
Coordinating a review team
Umbrella reviews require at least two independent reviewers for screening, extraction, and quality assessment. Coordinating tasks, tracking progress, and resolving disagreements across a team — especially when team members are at different institutions or in different time zones — adds logistical complexity on top of the methodological work.
Using a collaborative research workspace where team members can see task assignments, track review progress, and access shared reference libraries in real time makes this coordination significantly easier. ScholarDock's collaborative workspace is built for exactly this type of multi-person, multi-stage research workflow — connecting references, project tasks, and team communication in one place so nothing falls through the cracks.
Tools and frameworks every umbrella review team should know
Conducting a rigorous umbrella review requires familiarity with several key tools and reporting frameworks:
PROSPERO — the international register for systematic review protocols, including umbrella reviews. Registering your protocol here increases transparency and helps prevent duplication.
PRISMA 2020 — the standard reporting guideline for systematic reviews and meta-analyses. Follow the PRISMA checklist to ensure your umbrella review is reported completely and transparently.
PRISMA-P 2015 — the protocol-specific extension of PRISMA, used when writing and publishing your review protocol before conducting the review.
AMSTAR 2 — the most widely used tool for assessing the methodological quality of systematic reviews. Essential for the quality appraisal stage of your umbrella review.
JBI Manual for Evidence Synthesis (Chapter 10) — the Joanna Briggs Institute's comprehensive methodological guide for conducting umbrella reviews, covering everything from protocol development to evidence grading.
Rayyan and Covidence — screening tools that help automate the title-and-abstract and full-text screening stages, with features for independent reviewer blinding and conflict resolution.
ScholarDock — a research project and reference management platform that helps teams organize the full umbrella review workflow: storing and annotating source PDFs, managing extraction data, assigning team tasks, and keeping every reference connected to the project. When you are juggling dozens of systematic reviews and hundreds of cross-referenced primary studies, having a single structured workspace prevents the organizational chaos that derails so many review projects.
Who publishes umbrella reviews and where are they used?
Umbrella reviews have become increasingly common across medicine, public health, psychology, nursing, and education. They are published in high-impact journals such as BMJ Medicine, the British Journal of Sports Medicine, The Lancet, and specialty journals across nearly every clinical discipline.
They are used to:
Inform clinical practice guidelines — organizations like the WHO and national health agencies use umbrella reviews to assess the totality of evidence on interventions
Shape public health policy — umbrella reviews on topics like physical activity, nutrition, and mental health interventions provide the broad evidence base policymakers need
Identify research gaps — by mapping what has and has not been systematically reviewed, umbrella reviews highlight areas where new primary research or focused systematic reviews are needed
Support grant applications and research agendas — demonstrating comprehensive knowledge of the existing synthesized evidence base strengthens funding proposals
Publications using the umbrella review method have escalated significantly since 2015, driven by the sheer volume of systematic reviews being produced across all disciplines.
Start your umbrella review with the right foundation
An umbrella review is one of the most powerful tools in a researcher's evidence synthesis toolkit — but only when it is conducted rigorously, reported transparently, and organized effectively. The methodology demands careful protocol development, comprehensive searching, critical appraisal of included reviews, thoughtful synthesis, and honest assessment of overlap and limitations.
The biggest barrier to a successful umbrella review is rarely methodological knowledge — it is organizational complexity. Tracking dozens of systematic reviews, hundreds of cross-referenced primary studies, team task assignments, quality assessments, and extraction data across disconnected tools leads to errors, delays, and frustration.
If your research team is planning an umbrella review — or any large-scale evidence synthesis — ScholarDock brings your entire workflow into one connected workspace. From organizing your reference library and annotating source reviews to assigning team tasks and tracking project milestones, ScholarDock, a research project and reference management platform, keeps every layer of your review connected, discoverable, and under control.