🧠 Concept

1. Pre-Modern Roots: The 17th–19th Centuries

The impulse to aggregate knowledge is ancient, but the systematic aggregation of empirical evidence is a relatively modern scientific practice. Its earliest methodological stirrings can be traced to the 17th century, when physicians and natural philosophers began questioning whether individual observations or aggregated ones better supported therapeutic claims.

In 1753, the Scottish physician James Lind — best known for his scurvy trial — published A Treatise of the Scurvy, in which he systematically reviewed prior reports of scurvy treatments and applied what we would now recognize as an early form of evidence synthesis. Lind's work is often cited as the first documented attempt to aggregate trial-like observations systematically.

In the 19th century, the French mathematician Pierre-Simon Laplace and the astronomer Karl Friedrich Gauss developed the statistical machinery — least squares, normal distributions, weighted averages — that would eventually underpin meta-analytic pooling. Their contributions were not framed as "evidence synthesis" at the time, but they provided the mathematical foundation for later work.

2. The Birth of Meta-Analysis: Early 20th Century

The term meta-analysis was coined much later, but the technique of statistically combining results across studies emerged in the early 20th century. In 1904, Karl Pearson — founder of modern statistics — used what is effectively a meta-analytic method to combine correlation coefficients from smallpox vaccination studies. This is generally regarded as the first published meta-analysis, though Pearson did not use the term.

Through the early 20th century, agricultural and psychological researchers, particularly at Rothamsted Experimental Station and in educational psychology, developed techniques for combining estimates from independent experiments. These were largely technical innovations in statistics, not yet part of a broader methodological movement.

3. The Glass Era: Naming and Formalizing Meta-Analysis (1976)

The modern era of meta-analysis begins with Gene V. Glass, an educational psychologist who in 1976 published a landmark paper titled "Primary, Secondary, and Meta-Analysis of Research." Glass introduced the term meta-analysis — literally, "analysis of analyses" — and proposed that researchers systematically combine effect sizes across studies to yield a single quantitative summary.

Glass's work was transformative. It legitimized the quantitative synthesis of findings across studies and sparked a methodological revolution across the social, educational, and biomedical sciences. His approach was further refined by statisticians such as Larry Hedges (who developed Hedges' g), Ingram Olkin, and John Ioannidis in later decades.

4. The Archie Cochrane Era: From Technique to Movement (1972–1992)

While meta-analysis was developing on the statistical side, another movement was unfolding in clinical medicine. In 1972, the British epidemiologist Archie Cochrane published Effectiveness and Efficiency: Random Reflections on Health Services. In this brief but seminal book, Cochrane argued that:

• Health care decisions should be grounded in evidence, not expert opinion.
• Randomized controlled trials (RCTs) provide the most reliable evidence of treatment effects.
• Because RCTs are scattered across journals and languages, their results must be systematically aggregated.

Cochrane famously criticized his profession for failing to synthesize evidence, noting that obstetric practice ignored clear signals from accumulated trial data for decades. His 1989 book chapter "A Critical Review, with Particular Reference to the Medical Profession" became a call to arms for the emerging evidence-based medicine movement.

5. The Cochrane Collaboration and the Evidence Revolution (1992–Present)

In 1992, on the initiative of Iain Chalmers and others, the Cochrane Centre was founded in Oxford — renamed the Cochrane Collaboration in 1993. Its mission: to prepare, maintain, and disseminate systematic reviews of healthcare interventions. Cochrane established standards for conduct and reporting that remain foundational today.

Parallel movements emerged:

• Campbell Collaboration (2000) — parallel to Cochrane, but focusing on social interventions (education, criminal justice, social welfare).
• Joanna Briggs Institute (JBI, 1996) — originally focused on nursing and allied health; later expanded to methodological rigor across evidence types, including qualitative and mixed-methods synthesis.
• Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre, 1993) — established in the UK to support systematic reviews in education and social policy.
• Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Centers (1997) — institutionalized evidence synthesis within US healthcare policy.
• National Institute for Health and Care Excellence (NICE, 1999) — integrated systematic reviews into national guidance production.

6. The Reporting Standards Era (1996–Present)

As systematic reviews proliferated, so did variability in how they were conducted and reported. To promote transparency and reproducibility, a series of reporting standards emerged:

• QUOROM (1999) — Quality of Reporting of Meta-Analyses; the first formal reporting guideline.
• PRISMA (2009) — Preferred Reporting Items for Systematic Reviews and Meta-Analyses; replaced QUOROM and became the global standard.
• PRISMA 2020 — the current, expanded iteration; accommodates a wider range of review types and modern methodological developments.
• PRISMA-S (2021) — extension specifically for search reporting.
• PRISMA-ScR (2018) — extension for scoping reviews.
• PRISMA-P (2015) — for protocols.

7. The Living Evidence Movement (2015–Present)

The most recent methodological frontier is the living systematic review — a review that is continuously updated as new evidence emerges. Enabled by automation, machine learning, and digital platforms, living reviews address a persistent criticism of traditional systematic reviews: they can become outdated shortly after publication.

Today, systematic reviews are produced not only by academic teams but also by national and international health agencies, WHO, national guideline bodies, and evidence-synthesis platforms such as RevProHub — the platform on which this training is built.

8. Why This History Matters

Understanding the history of evidence synthesis is not nostalgia — it is methodological grounding. It explains:

• Why we have so many standards — each arose in response to specific problems.
• Why methodology matters — poor synthesis has caused real harm (e.g., delayed adoption of effective treatments, promotion of ineffective ones).
• Why the field is still evolving — the tools, standards, and platforms we use today are the result of decades of refinement, and the next decade will bring further change.

🧠 Concept

1. What Is an Evidence Hierarchy?

An evidence hierarchy is a framework that ranks research designs according to their ability to answer questions of treatment or intervention effectiveness without bias. The core intuition is straightforward: a rigorously conducted randomized trial provides more reliable evidence about a treatment's effect than an anecdote, expert opinion, or uncontrolled case series.

The hierarchy is not a statement that "lower" designs are worthless — different research questions require different designs. A well-conducted qualitative study answers questions about experiences and mechanisms; a cohort study answers questions about prognosis or rare harms. The hierarchy addresses a specific class of question: Does this intervention work, and by how much?

2. The Classic Pyramid

The most widely cited depiction is a pyramid, from bottom (lowest) to top (highest):

1. Ideas, editorials, opinions — expert reasoning without supporting data.
2. Animal research — provides mechanistic plausibility but cannot establish human efficacy.
3. In vitro research — establishes biological mechanism.
4. Case reports and case series — describe individual or small-group observations; useful for signal detection, not for causal inference.
5. Case-control studies — compare past exposures between cases and controls; efficient for rare outcomes but vulnerable to selection and recall bias.
6. Cohort studies — follow exposed and unexposed groups over time; strong for prognosis and harms but susceptible to confounding.
7. Randomized controlled trials (RCTs) — random allocation balances known and unknown confounders; the reference standard for efficacy questions.
8. Systematic reviews and meta-analyses of RCTs — the highest tier for treatment effectiveness questions.

3. Why Systematic Reviews Sit at the Top

A single RCT, however rigorous, can mislead. Systematic reviews address this by systematically identifying all relevant studies, appraising their quality, and synthesizing their results. Four reasons underpin their top position:

• Statistical power. Pooling data from multiple studies increases precision and enables detection of smaller but clinically meaningful effects.
• Replication. A single positive trial may be a false positive; concordant evidence from multiple trials strengthens causal inference.
• Bias mitigation. Systematic reviews assess risk of bias across studies and can account for publication bias.
• Transparency. A pre-specified protocol, reproducible search, and structured appraisal make conclusions verifiable.

4. Primary, Secondary, and Tertiary Evidence

5. Question-Specific Hierarchies

Modern methodologists recognize that the hierarchy is question-dependent. A single hierarchy does not apply to all inquiry. Alternative hierarchies include:

• Diagnostic questions: Cross-sectional studies with consecutive sampling and blinding, ideally synthesized via DTA systematic reviews.
• Prognostic questions: Cohort studies with long follow-up, ideally synthesized.
• Qualitative questions: In-depth interviews and focus groups, synthesized via meta-aggregation or meta-ethnography.
• Economic questions: Cost-effectiveness analyses built on systematic review evidence.
• Prevalence questions: Population-based cross-sectional studies, ideally synthesized.

6. Limitations and Misuse

Despite its usefulness, the hierarchy is frequently misused. Common errors include:

• Assuming RCTs are always feasible or ethical. For rare harms or long-term outcomes, RCTs may be impossible; well-conducted cohorts may be the best available evidence.
• Equating "top of the hierarchy" with "correct." A poorly conducted systematic review can be worse than a well-conducted RCT.
• Ignoring applicability. A rigorous review from a different population, setting, or healthcare system may not transfer.
• Confusing the hierarchy with certainty. GRADE explicitly separates study design from certainty of evidence.

🧠 Concept

1. From Evidence-Based Medicine to Evidence-Informed Policy

The term evidence-based medicine (EBM) — coined in the early 1990s — originally emphasized the integration of individual clinical expertise with the best available external evidence. Over time, the term was adapted to public health and policy as evidence-based policy. However, critics noted that this framing implies a linear, prescriptive relationship between evidence and decision-making — as if decisions could be read directly from the evidence.

Evidence-informed decision-making (EIDM) emerged as a more realistic alternative. EIDM acknowledges that decisions are influenced by multiple inputs:

• Best available evidence — systematic reviews, primary studies, program evaluations, surveillance data.
• Context — political priorities, resource constraints, cultural norms, health system capacity.
• Stakeholder values and preferences — patients, providers, communities, decision-makers.
• Practical considerations — timing, feasibility, cost, sustainability, equity.

The key shift is this: evidence informs rather than dictates decisions. Evidence-synthesis products — such as systematic reviews, policy briefs, and guidelines — are powerful inputs, but they must be interpreted within the realities of the systems they serve.

2. The EIDM Cycle

Most EIDM frameworks describe a cyclical process:

1. Identify and clarify the problem. Frame the question in policy-relevant terms.
2. Gather evidence. Use the best-available evidence, ideally from systematic reviews.
3. Appraise and synthesize. Assess quality, certainty, and applicability.
4. Engage stakeholders. Include decision-makers, implementers, and affected communities.
5. Make a decision. Weigh evidence, context, and values.
6. Implement. Translate the decision into action.
7. Monitor and evaluate. Assess outcomes; refine.

3. Where Systematic Reviews Fit

Systematic reviews occupy a pivotal position in the EIDM ecosystem. They serve as:

• The most reliable evidence on effectiveness — because they synthesize findings across settings and reduce the risk of cherry-picking.
• An honest broker of uncertainty — high-quality reviews explicitly report what is known, what is uncertain, and what gaps remain.
• A bridge to policy instruments — reviews feed into guidelines, policy briefs, and economic evaluations.

4. Knowledge Translation

Knowledge translation (KT) — sometimes called knowledge mobilization, dissemination, or implementation science — is the process of moving evidence from production into use. Key KT concepts:

• Push efforts: Researchers disseminate findings (publications, presentations, policy briefs).
• Pull efforts: Decision-makers actively seek evidence (commissioned reviews, evidence services).
• Exchange efforts: Researchers and decision-makers collaborate to co-produce and interpret evidence.
• Integrated KT: Decision-makers are engaged from the outset — from question formulation through to implementation.

5. Barriers and Facilitators

Research on EIDM has identified recurring obstacles:

6. EIDM in Low- and Middle-Income Countries

In low- and middle-income countries (LMICs), EIDM faces distinctive challenges:

• Fragmented health information systems
• Limited local research production
• Weak integration between research and policy institutions
• Restricted access to international databases and paywalled literature
• Competing priorities and scarce resources

However, LMICs also show promising innovations: institutionalized evidence-to-policy units, national health research agendas, and digital platforms. The WHO's Evidence-Informed Policy Network (EVIPNet) has supported such work across Africa, Asia, and Latin America.

🧠 Concept

1. What Is a Traditional Review?

A traditional review — also called a narrative review — is a synthesis of literature that follows no explicit, reproducible methodology for identifying, selecting, or appraising studies. The reviewer typically selects studies based on personal knowledge, editorial judgment, or thematic convenience. Traditional reviews can be valuable for:

• Providing an overview of a field for teaching or orientation.
• Exploring theoretical frameworks or conceptual debates.
• Raising hypotheses to be tested in primary research.
• Describing the historical evolution of a topic.

However, traditional reviews are not reliable sources of evidence on intervention effectiveness. Their methodology is not designed to control for bias, and their conclusions are not independently verifiable.

2. What Is a Systematic Review?

A systematic review is a synthesis that follows a pre-defined, transparent, and reproducible methodology to identify, appraise, and synthesize all relevant studies addressing a specific question.

Key structural elements include:

• A pre-specified protocol — registered in advance (e.g., PROSPERO).
• A comprehensive search strategy — designed to identify all eligible studies, including unpublished ones.
• Explicit eligibility criteria — determined before screening begins.
• Dual, independent screening — reducing selection bias.
• Systematic data extraction — using piloted, structured forms.
• Critical appraisal — using standardized risk-of-bias tools.
• A planned synthesis — quantitative (meta-analysis) or qualitative (narrative synthesis).
• Transparent reporting — following PRISMA 2020.

3. Comparative Differences

4. Risks of Traditional Reviews

Because they lack transparent methodology, traditional reviews are vulnerable to several biases:

• Selection bias — cherry-picking studies that support the reviewer's prior beliefs.
• Publication bias — over-relying on positive or famous findings.
• Language bias — citing only English-language papers.
• Confirmation bias — interpreting ambiguous evidence in line with pre-existing views.
• Irreproducibility — a different reviewer cannot replicate the same conclusions from the same literature.

Empirical research has shown that traditional reviews frequently disagree with subsequent systematic reviews on the same topic — and in the case of some therapies (e.g., early reviews of acupuncture for chronic pain), traditional reviews were considerably more optimistic than rigorous systematic assessments.

5. When Is Each Approach Appropriate?

Traditional review is acceptable when:

• Providing conceptual overviews for teaching purposes.
• Scoping a topic before a systematic review.
• Discussing theory, philosophical concepts, or historical context.
• Raising questions, not answering them.

A systematic review is required when:

• Answering a question about intervention effectiveness, diagnostic accuracy, or prognosis.
• Supporting a policy or clinical guideline.
• Informing programmatic decisions about resource allocation.
• Producing a publishable review in a high-impact journal.
• Contributing to a Cochrane, JBI, or Campbell review.

6. Hybrid Forms — Systematic Reviews with Narrative Elements

Modern practice recognizes hybrid forms:

• Systematic review with narrative synthesis — when meta-analysis is not possible, but systematic methodology is still applied (SWiM reporting guidelines).
• Scoping reviews — systematic approach to mapping literature on broad topics.
• Rapid reviews — systematic approaches with pragmatic methodological shortcuts, transparently reported.
• Integrative reviews — combine quantitative and qualitative evidence systematically.

🧠 Concept

1. Why Multiple Review Types Exist

A systematic review is not a single, monolithic product. Its methods must match its question. A review asking "Does drug X reduce mortality?" requires very different methods from one asking "What are patients' experiences of living with chronic pain?" or "How accurate is test T for detecting disease D?"

This chapter introduces the five most common families of systematic review, their typical questions, and their methodological hallmarks.

2. Intervention Reviews

Question: Does intervention X work for population P in comparison to C, on outcome O?

Study designs: RCTs primarily; non-randomized studies if RCTs are unavailable or inappropriate (harms, rare outcomes).

Typical tools: RoB 2 (RCTs), ROBINS-I (non-randomized studies), GRADE for certainty.

Reporting standard: PRISMA 2020.

3. Diagnostic Test Accuracy (DTA) Reviews

Question: How accurately does test T detect condition C in population P, compared to reference standard R?

Study designs: Cross-sectional or cohort studies with consecutive or random sampling, blinded comparison to the reference standard.

Typical tools: QUADAS-2 for quality, bivariate or HSROC models for meta-analysis of sensitivity/specificity.

Key metrics: Sensitivity, specificity, positive and negative likelihood ratios, diagnostic odds ratio, AUC.

Reporting standard: PRISMA-DTA.

4. Prognostic Reviews

Question: What is the likelihood of outcome O in population P over time T, based on factor F?

Study designs: Cohort studies; sometimes nested case-control.

Typical tools: QUIPS (Quality in Prognosis Studies), CHARMS for data extraction.

Key metrics: Hazard ratios, risk ratios, incidence rates, survival curves.

Reporting standard: PRISMA 2020 with prognostic extensions.

5. Qualitative Evidence Synthesis (QES)

Question: How do people experience phenomenon X? What are their perceptions, needs, or barriers?

Study designs: Interviews, focus groups, ethnographic and observational studies; qualitative component of mixed-methods studies.

Typical methods: Meta-aggregation (JBI), meta-ethnography, thematic synthesis, framework synthesis.

Typical tools: JBI Critical Appraisal Checklist for Qualitative Research, CASP Qualitative Checklist, CERQual for confidence.

Reporting standards: ENTREQ (for qualitative evidence synthesis); PRISMA 2020 with qualitative extensions.

6. Mixed-Methods Systematic Reviews (MMSR)

Question: Combines multiple types of evidence — e.g., does the intervention work, and why or how does it work?

Design: Two or more syntheses integrated. Common integration designs:

• Convergent — quantitative and qualitative syntheses conducted in parallel and merged.
• Sequential explanatory — quantitative synthesis first, then qualitative to explain findings.
• Sequential exploratory — qualitative first, then quantitative to test hypotheses.

Methodological reference: JBI Mixed Methods Systematic Review methodology, Lizarondo et al. 2020.

7. Scoping Reviews, Rapid Reviews, and Other Relatives

These are close relatives of systematic reviews and are often confused with them:

• Scoping review — maps the extent, range, and nature of evidence on a topic; does not usually assess quality; uses PRISMA-ScR.
• Rapid review — systematic methodology with pragmatic acceleration (limited search, single reviewer for some steps); must be transparently reported.
• Umbrella review — systematic review of existing systematic reviews; uses AMSTAR 2 for quality.
• Overview of reviews — similar to umbrella; focuses on mapping higher-level evidence.
• Living systematic review — continuously updated as new evidence appears.

8. Matching Review Type to Question