# Lesson 12: Integrated Appraisal of Epidemiological Research

*Companion-podcast transcript • Sarah & Kiffer*  
*Office Hours episode to listen to after working through the lesson*

---

**Sarah:** Welcome back to Office Hours. I'm Sarah.

**Kiffer:** And I'm Kiffer. This episode goes with Lesson twelve, Integrated Appraisal of Epidemiological Research, which is the last lesson of the course.

**Sarah:** It's meant for after you've finished the lesson. We'll add some perspective on how appraisal works in practice and some critique of the tools themselves, work through the questions students tend to find hardest with this material, and do some extra worked examples, including a couple that are harder than the ones on the lesson page.

**Kiffer:** Along the way there are a few places where we'll ask you to work something out before we give the answer. You'll hear a few seconds of quiet at each one, and you can pause the audio if you'd like more time to think it through.

**Sarah:** We'll cover four questions and one disagreement. First, how much a complete, well-reported paper should raise your confidence. Second, why a long list of results makes any one significant result less convincing. Third, how strong a hidden confounder would need to be to explain a finding away. Fourth, what happens when you pool studies that disagree. After that, Kiffer and I will argue about the hierarchy of evidence.

**Kiffer:** Question one. Suppose a paper ticks every STROBE item. The design is named in the title, every variable is defined, the flow diagram is there, and the limitations get a full paragraph. How much should all of that raise your confidence in its conclusion?

**Sarah:** Some, because you can finally see what the authors did. The distinction between reporting quality and methodological quality matters here. A complete report lets you judge the study, and it can also describe a flawed study in perfect detail.

**Kiffer:** Let's make that concrete with a made-up paper. A hypothetical cohort of eight thousand office workers in a Canadian city asks whether cycling to work lowers the risk of type 2 diabetes over ten years. The report is complete and clearly written. The authors adjust for age, sex, income, diet and body weight, and the body weight they use is the one measured at the end of follow-up. They find a hazard ratio of 0.95, with a confidence interval that crosses one, and they conclude that cycling has no meaningful effect on diabetes.

**Sarah:** So here's the question for everyone. Which link in the seven-step inferential chain breaks in this study, and what does that do to the estimate? Take a few seconds.

*(Pause)*

**Kiffer:** The causal model breaks. Cycling plausibly lowers body weight, and lower body weight lowers diabetes risk. Weight measured at the end of follow-up sits on the pathway between the exposure and the outcome, so it's a mediator. Adjusting for it removes the part of the effect that works through weight. What's left describes something narrower than the total effect the authors say they're estimating, and it's pulled toward the null.

**Sarah:** And the complete report is what let us find that. Because the authors told us exactly which weight measurement they used, a careful reader can see the problem in the methods section.

**Kiffer:** That's the right way to hold the two ideas together. STROBE asks authors to say how they chose their confounders, and here a careful reader can see that one of them was chosen badly. A better analysis would adjust for weight at baseline, measured before follow-up began, and would show a causal diagram explaining why.

**Sarah:** It's also a good example of why the order of the chain matters. The causal model is the third decision, well before the analysis. Everything after it, the tidy confidence interval and the careful discussion, inherits the problem, and a reader who jumps straight to the results table would never notice.

**Kiffer:** Question two is about multiple testing, and Sarah, I'd like you to work this one aloud. A hypothetical trial of a school nutrition program reports one primary outcome and twelve secondary outcomes, each tested at the usual 0.05 level with no correction. Suppose the program truly does nothing for any of them. What's the chance that at least one of the twelve secondary outcomes comes out significant anyway? Everyone listening, try it too. Take a few seconds.

*(Pause)*

**Sarah:** Each test has a five percent chance of a false positive, and there are twelve tests. Twelve times 0.05 is 0.6, so there's a sixty percent chance.

**Kiffer:** Let's test that method before we accept it. What would the same method give you if the trial had reported twenty-five secondary outcomes?

**Sarah:** Twenty-five times 0.05 is 1.25, which would be a hundred and twenty-five percent. That's impossible, so the method is wrong. I think what I calculated is the expected number of false positives. Across twelve tests you'd expect about 0.6 of one, and an expected count is allowed to go above one.

**Kiffer:** That's exactly what it is. So how do you get the probability of at least one false positive?

**Sarah:** Work out the chance that every test comes out clean, and subtract it from one. Each test is clean with probability 0.95, so all twelve are clean with probability 0.95 to the power of twelve, which is about 0.54. One minus 0.54 is about 0.46. So there's roughly a forty-six percent chance of at least one false positive.

**Kiffer:** That's correct, assuming the twelve tests are independent. Health outcomes are often correlated with each other, and positively correlated tests give a somewhat lower chance, but it stays far above five percent.

**Sarah:** In fairness to my first attempt, multiplying works well enough for two or three tests, doesn't it?

**Kiffer:** It does. For two tests, multiplying gives ten percent, and the exact answer is 9.75 percent. The two methods drift apart as the number of tests grows. For appraisal, the practical message is that a single p value of 0.04 among many unplanned comparisons is a lead to follow up, and the pre-registered primary outcome carries most of the weight.

**Sarah:** Which is why pre-registration is on the list of quality indicators. If the primary outcome was written down before the data came in, you know which test the study was built to run, and you can read everything else as exploratory.

**Kiffer:** And if the published primary outcome differs from the registered one, with no explanation, that's one of the clearest red flags you'll find in a trial report, because it suggests the authors chose the headline after seeing the results.

**Sarah:** Question three. How strong would a hidden confounder need to be to explain a result away? This is the E-value, which the lesson lists among the sensitivity analyses. Kiffer, can you give the formula in words, since it goes a little beyond the lesson?

**Kiffer:** For a risk ratio above one, the E-value is the risk ratio plus the square root of the risk ratio times the quantity risk ratio minus one. For a risk ratio below one, you first take one divided by the risk ratio and then use the same formula. You can also apply it to the confidence limit closest to one, which tells you how strong a confounder would need to be to pull the interval across to no effect.

**Sarah:** Here's our harder example, and it's made up. A hypothetical cohort finds that adults who take a daily multivitamin have a risk ratio of 0.75 for cardiovascular death, with a confidence interval from 0.62 to 0.90, after adjusting for age, sex, income and education. What's the E-value for the point estimate, and what's the E-value for the interval? There's a trap in the very first step. Take a few seconds.

*(Pause)*

**Kiffer:** The trap is putting 0.75 straight into the formula. The risk ratio minus one would be minus 0.25, and you'd be taking the square root of a negative number. So first invert it. One divided by 0.75 is four thirds, which is about 1.33.

**Sarah:** Then four thirds minus one is one third, and four thirds times one third is four ninths, about 0.44. The square root of four ninths is two thirds, about 0.67. Four thirds plus two thirds is exactly two, so the E-value for the point estimate is two.

**Kiffer:** Good. Now the interval. Which of the two limits do we use, and what do you get?

**Sarah:** The limit nearest one, which is 0.90. One divided by 0.90 is about 1.11. Then 1.11 minus one is 0.11, and 1.11 times 0.11 is about 0.12. The square root of that is about 0.35, and 1.11 plus 0.35 gives about 1.46.

**Kiffer:** Here's how to read those numbers. A hidden confounder associated with both multivitamin use and cardiovascular death by a risk ratio of about two each, over and above the measured confounders, could explain away the whole association. A confounder associated with each by about 1.46 could move the interval to include one.

**Sarah:** And is a confounder that strong plausible here?

**Kiffer:** That's where the second idea comes in, which is healthy user bias. People who take a daily vitamin tend to exercise more, smoke less and keep up with regular checkups. Taken together, that bundle of habits could plausibly be associated with cardiovascular death by a risk ratio of one and a half or more. So an E-value of 1.46 for the interval offers little reassurance.

**Sarah:** There's a real check on this, too. In the Physicians' Health Study II, a randomized trial of 14,641 male physicians in the United States followed for about eleven years, the men who took a daily multivitamin had no lower risk of major cardiovascular events than the men who took a placebo.

**Kiffer:** That fits what the E-value suggested. The E-value measures how much confounding would be needed to explain a result, and your subject knowledge then decides whether that much is believable. Here that much confounding was believable, and the trial result is consistent with it.

**Sarah:** One caution I'd add. The formula is built for risk ratios. If a paper reports an odds ratio for a common outcome, you'd need to convert it first, so check which kind of ratio you have before you use it.

**Kiffer:** Question four. Two studies disagree, and someone suggests pooling them in a meta-analysis to settle the matter. What does pooling actually do with studies of very different quality?

**Sarah:** The pooled estimate weights studies by inverse variance. Let's try that with made-up numbers. There are three hypothetical studies of the same exposure. Study one is a large online convenience survey, and its estimate has a standard error of 0.05. Study two is a mid-sized cohort with a standard error of 0.10. Study three is a small, careful cohort with a standard error of 0.20. What share of the total weight goes to Study one? Take a few seconds.

*(Pause)*

**Kiffer:** Each weight is one divided by the standard error squared. For Study one, 0.05 squared is 0.0025, and one divided by that is four hundred. For Study two, 0.10 squared is 0.01, which gives a weight of one hundred. For Study three, 0.20 squared is 0.04, which gives twenty-five. The total is five hundred and twenty-five, and four hundred divided by five hundred and twenty-five is about seventy-six percent.

**Sarah:** So the convenience survey supplies about three quarters of the pooled estimate, and the careful small cohort supplies less than five percent.

**Kiffer:** That's right. Inverse-variance weights reward precision, and precision comes mostly from size. If Study one carries a bias, say from crude self-report and a self-selected sample, the pooled result will mostly reproduce that bias, with a narrow confidence interval around it. That's the sense in which a meta-analysis can give a precise estimate of a biased number.

**Sarah:** Which is why risk of bias is handled separately from the weights. You'd run a sensitivity analysis that drops the studies at high risk of bias, or stratify the results by risk of bias, and see whether the answer changes.

**Kiffer:** And if dropping Study one moves the pooled estimate a long way, you've learned that the conclusion depends heavily on the study you trust least. A good review reports both results and says which one it believes.

**Sarah:** There's a real case that shows what happens when that kind of appraisal is skipped. Wakefield and colleagues' 1998 paper in The Lancet, which claimed a link between the measles, mumps and rubella vaccine and autism, was based on twelve children. With no comparison group, a case series that small couldn't support a causal claim at all.

**Kiffer:** And the problems a careful reader could see were only part of it. The Lancet retracted the paper in February 2010, and in 2011 the journalist Brian Deer reported in the BMJ that the paper had misrepresented the children's cases. The weaknesses in design were visible from the start, and a reader who had worked through the five stages would have given the paper very little weight.

**Sarah:** That brings us to the part where Kiffer and I disagree, which is the hierarchy of evidence. The glossary calls it a heuristic, and I want to defend it. For most readers, most of the time, knowing that a randomized trial protects against confounding better than a cohort study, and that a cohort study does better than a cross-sectional survey, gets you close to the right answer quickly.

**Kiffer:** I agree it's a reasonable starting point. My concern is that people treat a design label as a quality score. A trial that loses forty percent of its participants to follow-up, or that measures a subjective outcome without blinding, can be less trustworthy than a careful cohort study. The 2008 reanalysis of the Nurses' Health Study makes the point from the other direction.

**Sarah:** Remind everyone what happened there, because I think it actually supports my side.

**Kiffer:** Earlier observational studies of hormone therapy and coronary heart disease had disagreed with the Women's Health Initiative trial. In 2008, researchers reanalysed the Nurses' Health Study data using the trial's eligibility criteria and timing, comparing women who started hormone therapy with women who didn't. The observational estimates came close to the trial's, and the earlier gap was largely explained by time since menopause and length of follow-up. So much of the problem lay in how the earlier observational analyses had been set up.

**Sarah:** And that's my point. It took a large trial to reveal that the original observational analyses were misleading. Without the trial, nobody would have known there was a gap to explain. The hierarchy earns its place because the errors it guards against are hard to see from inside a single observational study.

**Kiffer:** That's a fair point, and it's the strongest version of your argument. I'd still say that a student who has finished this course should be able to read past the label, and my view is that design tells you where to start while the appraisal tells you where to end up.

**Sarah:** That's close to how GRADE works, and the Canadian Task Force on Preventive Health Care uses GRADE to grade its recommendations. Under GRADE, evidence from randomized trials starts at high certainty and evidence from non-randomized studies traditionally starts at low. Evidence can be rated down for risk of bias, inconsistency, indirectness, imprecision or publication bias. Evidence from non-randomized studies can be rated up for a strong association, a dose-response gradient, or when plausible residual confounding would push against the observed effect.

**Kiffer:** So here's a practical rule we can share. Use the design as your starting estimate of how much to trust a study, move that estimate up or down according to what your appraisal of the particular study shows, and say which way you moved and why.

**Sarah:** I'm happy with that rule, as long as the starting point stays visible. I'd still want a strong, stated reason before I ranked a cross-sectional survey above a well-run trial on the same question.

**Kiffer:** Let's pull it together with three things to take away from this lesson, and from the course as a whole.

**Sarah:** First, a complete report is what makes appraisal possible, and it says nothing on its own about whether the study is sound. Check every link of the chain, and especially the causal model, because a well-reported study can still adjust for the wrong variables.

**Kiffer:** Second, read any single significant result against everything else the study tested, and against how strong a hidden confounder would need to be to explain it. The chance of at least one false positive and the E-value both give you a number to reason with.

**Sarah:** And third, when you combine studies, precision and trustworthiness are separate questions. The pooled estimate follows the largest studies, so check whether the answer holds when you set aside the studies at high risk of bias.

**Kiffer:** If you'd like more practice, rework the multivitamin E-value example from this episode with a risk ratio of 0.60 and an upper confidence limit of 0.80, and then ask whether healthy user bias could plausibly reach that strength.

**Sarah:** This is the last lesson of the course, so this is also our last Office Hours episode for this course. Thank you for working through all twelve lessons with us.

**Kiffer:** Everything in this course comes back to one habit, which is asking where a study's reasoning is weakest and how much that weakness matters. Take care, everyone.

**Sarah:** Good luck with all the reading you do from here, and goodbye for now.
