# Lesson 2: Systematic Reviews and Meta-Analysis

*Companion-podcast transcript • Sarah & Kiffer*  
*Office Hours episode to listen to after working through the lesson*

---

**Sarah:** Welcome back to Office Hours. I'm Sarah.

**Kiffer:** And I'm Kiffer. This episode goes with Lesson two, Systematic Reviews and Meta-Analysis.

**Sarah:** It's meant for after you've worked through the lesson. We'll add some perspective on how reviews get used, some critique of the methods, a look at the questions students tend to find thorny with this material, and a few extra worked examples, including some that are harder than the ones on the lesson page.

**Kiffer:** There are a few places where we'll ask you to work something out before we give the answer. When that happens, you'll hear a few seconds of quiet, and you can pause the audio if you'd like more time to think it through.

**Sarah:** We'll take five questions. First, why a systematic review sits at the top of the evidence pyramid when it can still be wrong, and what a search actually does. Second, how a carefully reported review can still earn the lowest AMSTAR 2 rating. Third, how a meta-analysis decides how much each study counts. Fourth, what I squared tells you and what it leaves out. And fifth, whether the random-effects model is always the safer choice, which is where Kiffer and I disagree a little.

**Kiffer:** Question one. If a systematic review can be badly done, why do we put it at the top of the pyramid at all? I think the answer is that the pyramid ranks designs by how well they can guard against bias and chance, and a review that pools every relevant trial can do that better than any one trial. That ranking describes what the design is capable of. Whether a given review lives up to it is a separate question, and that's what appraisal is for.

**Sarah:** So the pyramid tells you where to look first, and tools like AMSTAR 2 tell you whether what you found deserves your trust.

**Kiffer:** Right. And there's a good historical reason the field built all this machinery. In 1992, Antman and colleagues showed that expert reviews of heart attack treatments lagged years behind what a cumulative meta-analysis of the trials had already shown.

**Sarah:** There's a nice picture of that idea in Cochrane's logo. It's a forest plot from the Cochrane review of corticosteroids given to women at risk of preterm birth. Each horizontal line is one trial, and the diamond sits clearly on the side that favours treatment. Cochrane says the review helped increase use of the treatment, which has probably saved thousands of premature babies.

**Kiffer:** And the same synthesis machinery shapes Canadian guidance. The Canadian Task Force on Preventive Health Care grades its recommendations with GRADE, so when you read one of its screening recommendations, the certainty rating behind it came from exactly the process in this lesson.

**Sarah:** I want to spend a minute on the search, because I think students underestimate how much can go wrong there. A review is only as complete as the search that fed it.

**Kiffer:** Let's do a made-up example. Suppose a team is reviewing whether walking groups reduce loneliness in older adults. In a hypothetical database, the word loneliness appears in nine hundred records, the phrase social isolation appears in fourteen hundred, and three hundred records contain both. A new searcher treats these as one concept but joins them with AND. How many records does that search return, and how many would OR have returned? Take a few seconds.

*(Pause)*

**Sarah:** AND keeps only records that contain both terms, so that's three hundred. OR keeps records with either term. If I just add nine hundred and fourteen hundred, I get twenty-three hundred, but that counts the three hundred overlapping records twice. So I take the three hundred away once, and OR returns two thousand.

**Kiffer:** That's right. Using AND between synonyms cut the search from two thousand records to three hundred, and nothing on the screen warns you. The rule from the lesson is OR within a concept and AND between concepts. A dated search log is what lets someone else spot a slip like that later.

**Sarah:** And even a perfect search only gets you to the knowledge layer of the DIKW pyramid. Suppose a review shows that walking groups reduce loneliness on average. A health authority still has to decide whether its own older residents can get to a walking group, who would run it, and who might be left out.

**Kiffer:** Yes, and I think that's the most useful thing the DIKW picture adds. The review supplies the best available knowledge, and the decision about a particular community still needs local context, values and judgement, which is the wisdom layer.

**Sarah:** Question two. How can a review that's carefully reported still earn the lowest AMSTAR 2 rating? I'll work through a made-up review, since this is the kind of appraisal students find confusing.

**Kiffer:** Here's the hypothetical review. It looks at school nutrition programs and childhood obesity. It follows the PRISMA 2020 checklist item by item, with a clear flow diagram. It lists every excluded study with a reason, uses a random-effects model with a justification, and discusses publication bias. But the authors searched only one database, with no trial registries or grey literature, and they never assessed the risk of bias of the included studies. What's your rating?

**Sarah:** The PRISMA reporting is strong, so my first instinct is to rate it fairly well. But PRISMA is about how a review is reported, and AMSTAR 2 is about how it was done. So I go through the critical domains. Searching a single database fails the item on the adequacy of the search, which is item four. Having no risk-of-bias assessment fails item nine. Those are two critical flaws.

**Kiffer:** Good. And with two critical flaws, what overall confidence rating does that give the review?

**Sarah:** More than one critical flaw means critically low. It doesn't matter how good the other items are, since AMSTAR 2 asks for an overall judgement and never adds up a score. A clear report of a weak method still leaves the method weak.

**Kiffer:** Exactly. And there's a practical consequence. The review's pooled estimate could look very precise, with a narrow confidence interval, and none of that precision protects you from studies the search missed or from bias inside the studies it found.

**Sarah:** Question three. How does a meta-analysis decide how much each study counts? I'd like to try this one myself, because I remember getting it wrong the first time.

**Kiffer:** Here are three made-up trials of a community salt-reduction program, each reporting the change in systolic blood pressure, in millimetres of mercury. Trial A reports a drop of four, with a standard error of one. Trial B reports a drop of eight, with a standard error of two. Trial C, the largest, reports a drop of two, with a standard error of 0.5. What's the fixed-effect pooled estimate? Take a few seconds.

*(Pause)*

**Sarah:** Okay. Inverse weighting, so each trial's weight is one divided by its standard error. That gives weights of one, one half and two, which add to three and a half. Then I multiply each drop by its weight. Four times one is four, eight times a half is four, and two times two is four. That's twelve, and twelve divided by three and a half is about 3.4. So the pooled drop is about 3.4.

**Kiffer:** Before we go on, do a quick sanity check. Which of the three trials should the pooled answer sit closest to, and why?

**Sarah:** Trial C, because it's the most precise. And 3.4 is a fair way from two. Hmm. Oh, I see it. The weight is one divided by the variance, and the variance is the standard error squared. I divided by the standard error itself.

**Kiffer:** That's the slip, and it's a very common one. Try it again with the variances, and see where the answer lands.

**Sarah:** The variances are one, four and a quarter, so the weights are one, a quarter and four. They add to 5.25. Four times one is four, eight times a quarter is two, two times four is eight. That sums to fourteen, and fourteen divided by 5.25 is about 2.67. So the pooled drop is about 2.7, and Trial C now carries about three quarters of the total weight.

**Kiffer:** Good, and the sanity check did the work there. Squaring the standard error matters a lot. Halving a standard error makes a study's weight four times larger, which is why one large trial can dominate a pooled estimate. The standard error of the pooled result is one divided by the square root of 5.25, about 0.44, so the ninety-five percent confidence interval runs from roughly 1.8 to 3.5.

**Sarah:** Which leads to the leave-one-out idea from Section four. If Trial C carries three quarters of the weight, what happens to the pooled estimate when we drop it? Take a few seconds.

*(Pause)*

**Kiffer:** Without Trial C, the weights are one and a quarter, which add to 1.25. Four times one plus eight times a quarter gives six, and six divided by 1.25 is 4.8. So the pooled drop jumps from about 2.7 to 4.8, close to double. That's an influential study by any standard.

**Sarah:** And the response to that isn't to throw Trial C out.

**Kiffer:** Correct. A big, careful trial is supposed to carry a lot of weight. The leave-one-out result tells you to look hard at Trial C, at its population, its dose and its risk of bias, and to report clearly how much the answer depends on it.

**Sarah:** Question four. What does I squared actually tell you? Let's stay with the same three trials. The trials clearly disagree, since they report drops of four, eight and two.

**Kiffer:** So let's compute Cochran's Q. It's the weighted sum of squared distances from the pooled estimate. Trial A is about 1.33 units from 2.67, and 1.33 squared times its weight of one is about 1.78. Trial B is about 5.33 away. Squared, that's about 28.4, and times a weight of a quarter gives about 7.11. Trial C is about 0.67 away, which squares to about 0.44, and times a weight of four gives about 1.78. Add them up and Q is about 10.67.

**Sarah:** With three trials, the degrees of freedom are two. So I squared is Q minus two, divided by Q. That's 8.67 divided by 10.67, which is about 0.81, or eighty-one percent. That's high by the usual benchmarks.

**Kiffer:** Now here's the harder version. Imagine a second, made-up meta-analysis. Three trials report exactly the same drops, four, eight and two. The only change is that every trial is a quarter of the size, so every standard error doubles, to two, four and one. What happens to the pooled estimate, and what happens to I squared? Take a few seconds.

*(Pause)*

**Sarah:** Doubling a standard error quarters the weight, so the weights become a quarter, a sixteenth and one. They've all shrunk by the same factor, so the pooled estimate is still about 2.67. And Q is built from those weights, so Q is a quarter of what it was, about 2.67. Then I squared is 2.67 minus two, divided by 2.67, which is about twenty-five percent.

**Kiffer:** So the effects are identical, the spread between them is identical, and I squared fell from eighty-one percent to twenty-five.

**Sarah:** That's the trap. Someone reading the second meta-analysis would say the trials agree fairly well, and they don't agree any better than in the first one. The trials are just too imprecise to show the disagreement clearly.

**Kiffer:** Exactly. I squared is the share of the observed variation that goes beyond chance, and how much variation chance produces depends on how precise the studies are. Rücker and colleagues showed in 2008 that as studies become more precise, I squared climbs toward a hundred percent, even when the between-study variance stays about the same. So a large I squared in a meta-analysis of big trials may describe differences that matter very little in practice.

**Sarah:** Which is why tau squared is worth reading alongside it, since it estimates the spread of the true effects in the units of the outcome itself. So what about the twenty-five, fifty and seventy-five percent benchmarks?

**Kiffer:** Higgins and colleagues proposed them in 2003, and they were careful to call them tentative. The Cochrane Handbook gives overlapping ranges, for example thirty to sixty percent as possibly moderate and fifty to ninety as possibly substantial, and it warns that thresholds can mislead. My own advice is to treat any benchmark as a prompt to look at the forest plot and ask whether the differences between studies would change a decision.

**Sarah:** That connects to GRADE, too. Inconsistency is one of the domains that can lower the certainty of the evidence. If our three salt trials were the whole evidence base, would the eighty-one percent alone be enough to downgrade?

**Kiffer:** I'd look at more than the number. All three trials show a drop in blood pressure, so they agree on the direction, and they differ on the size. A guideline panel would ask whether a drop of two and a drop of eight lead to different decisions. If they do, and nothing in the protocol explains the gap, that unexplained inconsistency is a reason to rate the certainty down. With only three trials, the panel would also have little room to explore the cause with a subgroup analysis.

**Sarah:** Question five. Is the random-effects model always the safer choice? I'll make the case for it first. The lesson's position is that random effects is generally preferred, because a single true effect across different populations and settings is rarely believable. In our salt example, I'd never assume three community programs have exactly the same effect.

**Kiffer:** I agree with that as a starting point. My hesitation is about what random effects does to the weights. Adding tau squared to every study's variance evens the weights out, so small studies gain influence relative to large ones. The Cochrane Handbook says this directly. If small studies differ systematically from large ones, through publication bias or poorer conduct, a random-effects analysis will make the effect of that bias worse.

**Sarah:** Is there a real case where that mattered?

**Kiffer:** The classic one is intravenous magnesium after a heart attack. Several small trials appeared to show a benefit, and a meta-analysis that combined them with a trial called LIMIT-2 suggested that magnesium reduced deaths. Then ISIS-4, which enrolled 58,050 patients, found no reduction in mortality. A later Cochrane review of twenty-six trials concluded that magnesium is unlikely to reduce deaths, and it flagged likely publication bias.

**Sarah:** But I'd say that example is mostly about publication bias, and no choice of model fixes missing studies. A fixed-effect analysis would have been dominated by ISIS-4, and that's good here only because ISIS-4 happened to be right. A fixed-effect model applied to a set of truly varied studies still gives a confidence interval that's too narrow, and I think that's the more common mistake in public health, where programs and populations really do differ.

**Kiffer:** That's fair, and I'd keep random effects as the default when studies differ in population, dose or setting, as the lesson recommends. What I'd add is a check. When the fixed-effect and random-effects estimates differ a lot, the gap is telling you something, usually that the small studies are pulling in a different direction from the large ones. That's a reason to look at the funnel plot and the small trials themselves before you trust either number.

**Sarah:** So we agree on a practical rule. Pre-specify the model in the protocol and justify it, run the other model as a sensitivity analysis, and treat a large gap between the two as a signal to investigate small-study effects.

**Kiffer:** Let's pull it together with three things to take away from this episode on reviews and meta-analysis.

**Sarah:** First, the evidence pyramid tells you where to look, and appraisal tells you whether to trust what you found. Good reporting and good conduct are separate things, and AMSTAR 2 judges the conduct.

**Kiffer:** Second, studies are weighted by one divided by the variance, which is the standard error squared. That squaring is why one precise trial can dominate the pooled estimate, and why a leave-one-out check is worth running every time.

**Sarah:** And third, I squared depends on how precise the studies are as well as on how much they disagree, so read it alongside tau squared and the forest plot, and treat the benchmarks as rough guides.

**Kiffer:** If you'd like more practice, rework the salt-reduction example with all three standard errors halved, and before you calculate anything, predict what happens to the pooled estimate and to I squared.

**Sarah:** Next time, it's Lesson three, Introduction to Observational Studies, where we go back down the pyramid to the designs that produce the studies a review pools.

**Kiffer:** Take care, everyone.

**Sarah:** See you in Lesson three.
