# Lesson 1: Foundations of Epidemiology

*Companion-podcast transcript • Sarah & Kiffer*  
*Office Hours episode to listen to after working through the lesson*

---

**Sarah:** Welcome back to Office Hours. I'm Sarah.

**Kiffer:** And I'm Kiffer. This episode goes with Lesson one, Foundations of Epidemiology.

**Sarah:** It's meant for after you've worked through the lesson. We'll bring in some perspective on where these ideas come from, some critique of how they get used, a few of the questions students tend to find thorny, and some extra worked examples, including one or two that are harder than anything on the lesson page.

**Kiffer:** At a few points we'll ask you to work something out before we give the answer. After the question there will be a few seconds of quiet, and you can pause the audio if you'd like more time to think it through.

**Sarah:** We have four questions. First, what made John Snow's comparison of two water companies so convincing, and why is he the person we remember? Second, if a study looks at many outcomes, how likely is a false alarm? Third, does a statistically significant result mean a finding is probably true? And fourth, what should count as evidence, and where does the hierarchy of evidence fit? That last one is where Kiffer and I see things a little differently.

**Kiffer:** Question one, then. Most people know the Broad Street pump story, with the map and the pump handle. I'd like to spend time on a second piece of Snow's work from the same epidemic, because it shows the logic of epidemiology more clearly than the map does.

**Sarah:** This is the comparison of the two water companies that supplied the same part of London.

**Kiffer:** Yes. Two private companies piped water into the same neighbourhoods south of the river. In 1852, one of them moved its intake upstream, away from the sewage. The other kept drawing water from a stretch of the river that received the city's sewage. Snow wrote that the pipes of each company went down all the streets, and that each company supplied both rich and poor, both large houses and small.

**Sarah:** So the households were mixed together street by street, and the main thing that differed between them was where their water came from.

**Kiffer:** That's the key point. Nobody assigned the water supply at random, yet it was sorted almost as if somebody had. Today we'd call that a natural experiment.

**Sarah:** Let's put numbers on it. These are Snow's own figures for the first seven weeks of the 1854 epidemic, up to late August. The downstream company supplied about forty thousand houses, and those houses had twelve hundred and sixty-three cholera deaths. The upstream company supplied about twenty-six thousand houses, which had ninety-eight deaths. How many times higher was the death rate in houses served by the downstream company? Take a few seconds.

*(Pause)*

**Kiffer:** Start with a common denominator, deaths per ten thousand houses. For the downstream company, twelve hundred and sixty-three divided by forty thousand and forty-six, times ten thousand, is about three hundred and fifteen. For the upstream company, ninety-eight divided by twenty-six thousand one hundred and seven, times ten thousand, is about thirty-seven and a half.

**Sarah:** And three hundred and fifteen divided by thirty-seven and a half is about 8.4. So houses on the downstream supply had roughly eight times the cholera death rate.

**Kiffer:** Right. Snow's printed table rounds the second figure down to thirty-seven, which gives about eight and a half, so the conclusion is the same either way.

**Sarah:** I'd like to push on the denominator, though. He counted houses. A house with twelve people in it and a house with two people count the same.

**Kiffer:** That's a fair criticism, and it's the kind of question worth asking of any rate. Houses are a rough stand-in for people at risk. The reason the comparison still holds up is that both companies served similar mixes of housing in the same streets, so there's no obvious reason one company's houses would be far more crowded. A difference of eight times is also too large to come from house size alone.

**Sarah:** What would have made you doubt the result?

**Kiffer:** If the downstream company had mainly served the poorest and most crowded streets, then poverty and crowding could explain some of the gap, and the water would get credit it didn't deserve. That's confounding, and it comes up in nearly every lesson in this course. It's why Snow's description of how the two supplies were mixed together matters as much as his death counts.

**Sarah:** Now the second half of the question. Why is Snow the one we remember?

**Kiffer:** Part of the answer is that his work was very good. But the network view from the lesson helps here. Snow worked from the weekly death returns that William Farr's office published, so the registration system was doing much of the counting. And historians like Jim Downs point to earlier investigators working in colonial settings. A British naval surgeon named James McWilliam investigated a yellow fever outbreak on the island of Boa Vista, in Cape Verde, in the mid-1840s, several years before Broad Street.

**Sarah:** So the method has a longer and less tidy history than the "father of epidemiology" title suggests.

**Kiffer:** Yes. And who gets remembered says something about whose work the field chose to record.

**Sarah:** Question two moves from history to how research goes wrong. The lesson separates outright fraud from questionable research practices, and the second group is much more common. I'd like to understand how an honest researcher ends up with a false finding.

**Kiffer:** Here's a made-up study. A team tests whether a new school program improves student health. Suppose the program truly does nothing. The team measures ten different outcomes, things like sleep, mood, physical activity and sick days, and tests each one at the usual five percent level. If the outcomes are independent, what's the chance that at least one comes out statistically significant just by chance? Take a few seconds.

*(Pause)*

**Sarah:** Let me try this one. Each test has a five percent chance of a false positive, and there are ten tests, so ten times five percent. That's fifty percent.

**Kiffer:** Before you settle on that, Sarah, try the same method with twenty-one outcomes and see what you get.

**Sarah:** Twenty-one times five percent is a hundred and five percent. Okay, that can't be right. A probability can't be more than one.

**Kiffer:** So where does adding the five percents together go wrong in this problem?

**Sarah:** Adding counts the studies where two or three outcomes are false positives more than once. The easier route is through the complement. The chance a single test is not significant is 0.95. For all ten to be not significant, I multiply 0.95 by itself ten times, which is about 0.60. So the chance of at least one false positive is one minus 0.60, about forty percent.

**Kiffer:** That's right. And with twenty outcomes it's about sixty-four percent. Adding gave a number that was too high, and for many outcomes it gives an impossible one, which is the sanity check that caught it.

**Sarah:** Forty percent is still huge. If the team reports only the outcome that worked, a reader sees one clean result.

**Kiffer:** That's selective outcome reporting, and the arithmetic shows why it matters. In practice the outcomes are usually correlated, which lowers the figure somewhat. But there are other choices too, like which covariates to adjust for or whether to add more participants. In 2011, Simmons, Nelson and Simonsohn simulated a few of those choices together and found false-positive rates above sixty percent, with no true effect at all.

**Sarah:** And it isn't rare. In the 2026 survey of Portuguese researchers that the lesson discusses, nearly forty-six percent said they had developed hypotheses after seeing their results.

**Kiffer:** Which is why preregistration focuses on writing down the primary outcome before the data arrive. It turns ten chances to find something into one stated test, plus exploratory analyses that are labelled as exploratory.

**Sarah:** So as a reader, what can I actually check when I pick up a paper?

**Kiffer:** Look for a registration number, and if there is one, look up the registered primary outcome and compare it with the one in the abstract. Notice when the abstract leads with a secondary outcome, or with a change within one group, while the main comparison between groups was null. That pattern is called spin. And count the outcomes and subgroups in the tables. A long list with one significant result deserves the arithmetic we just did.

**Sarah:** Question three builds on that. When a paper reports a result with a p-value below 0.05, a lot of students read that as a ninety-five percent chance the finding is real. I used to read it that way myself.

**Kiffer:** It's one of the most common misreadings in the field. The five percent is the chance of a significant result when there is no true effect. The question a reader cares about runs the other way: given a significant result, how likely is it that the effect is real? That depends on how many of the hypotheses being tested were true to begin with, and on how well the studies can detect a real effect. That's the core of John Ioannidis's 2005 paper in PLOS Medicine.

**Sarah:** So the p-value answers the question, if there were no effect, how surprising would this result be? It doesn't tell us directly how likely the effect is to be real.

**Kiffer:** Exactly. To get from one to the other, you need to know something about the hypotheses a field is testing.

**Sarah:** Let's work it with made-up numbers.

**Kiffer:** Imagine a research field that tests one thousand hypotheses. Suppose one in ten is actually true, so one hundred are true and nine hundred are false. Each study has eighty percent power, which means an eighty percent chance of detecting a true effect, and uses the five percent significance level. Out of all the significant results, what share are true? Take a few seconds.

*(Pause)*

**Sarah:** The true ones first. Eighty percent of one hundred is eighty true positives. Then the false ones. Five percent of nine hundred is forty-five false positives. So there are one hundred and twenty-five significant results, and eighty of them are true. Eighty divided by one hundred and twenty-five is 0.64.

**Kiffer:** So about sixty-four percent, well short of ninety-five. Now suppose the studies are small, so the power drops to twenty percent.

**Sarah:** Then twenty percent of one hundred is twenty true positives, and there are still forty-five false ones. Twenty out of sixty-five is about thirty-one percent. So most of the significant findings would be false.

**Kiffer:** Here's the harder step, which combines this with question two. Suppose the field's studies have eighty percent power again, but every team has the kind of flexibility from our made-up school study, so a false hypothesis comes out significant about forty percent of the time.

**Sarah:** Forty percent of nine hundred is three hundred and sixty false positives. With eighty true positives, that's eighty out of four hundred and forty, about eighteen percent.

**Kiffer:** Good. There's one trap to check. The same flexibility also helps true hypotheses reach significance, so eighty true positives might be too few.

**Sarah:** But there are only one hundred true hypotheses, so at most one hundred true positives. One hundred out of four hundred and sixty is about twenty-two percent. Even in the best case, fewer than one in four significant findings would be real.

**Kiffer:** That's the argument in a few lines of arithmetic. The numbers are invented, and real fields differ a great deal. What the example shows is which conditions make false findings common: few true hypotheses, small studies, and flexible analysis.

**Sarah:** And it fits the epidemiology examples in Table 1.4 of the lesson, where observational findings on hormone therapy and beta-carotene didn't hold up in trials.

**Kiffer:** Those had another ingredient, confounding, which pushes results in a consistent direction. But the general lesson is the same. One significant study is a starting point, and replication is what makes a finding believable.

**Sarah:** Question four is about the hierarchy of evidence, with randomized trials at the top and qualitative work and lived experience near the bottom. The lesson argues that the hierarchy devalues other kinds of knowledge, and this is where Kiffer and I disagree a little.

**Kiffer:** Go ahead and set out your side first, Sarah, and then I'll give mine.

**Sarah:** My view is that the hierarchy does real harm when it's applied as a ranking of all evidence. If a funding panel or a guideline committee treats interviews with people who use drugs as weak evidence by default, it ignores what those people know about why a service works or fails. The lesson's point about standpoint theory is that people living with a problem often see parts of it that researchers miss. A single ranking can't capture that.

**Kiffer:** Can you give an example of the kind of knowledge a ranking like that would miss?

**Sarah:** Take a made-up harm reduction service that people stop using after a few weeks. A survey might show the drop-off. Peer workers who use the service might explain that the opening hours clash with when people actually need it, or that the security check at the door feels like policing. That knowledge points straight at the fix, and a ranking that places it near the bottom would tell a planner to discount it.

**Kiffer:** I agree with a good part of that. Where I'd push back is on dropping the hierarchy altogether. For one particular kind of question, whether an intervention changes an outcome, randomization deals with confounding in a way no other design can. The hormone therapy story is a case where a trial overturned a confident observational finding, and prescribing changed as a result.

**Sarah:** But the lesson also shows how the hierarchy gets used as a weapon. The tobacco industry demanded trial evidence that nobody could ethically produce.

**Kiffer:** That's the strongest version of your argument, and I accept it. Bradford Hill made a related point in 1965. He wrote that none of his nine viewpoints can bring indisputable evidence for or against causation, and that none of them is required. Smoking was shown to cause lung cancer without a trial, from consistent observational evidence across many studies. So I'd keep the hierarchy for questions about intervention effects and treat it as one input, with the full body of evidence judged on its merits.

**Sarah:** I'd go further than Kiffer in treating lived experience as evidence in its own right, but I think we agree on a practical rule. Name the question first, then ask which kind of evidence can answer it.

**Kiffer:** Let's try that rule on a made-up example. A hypothetical First Nation runs a community food program, and three questions come up. Did the program lower rates of type 2 diabetes? What does the program mean to the families who take part? And who decides how the data are stored and used? For each question, what kind of evidence or framework fits? Take a few seconds.

*(Pause)*

**Sarah:** The first is a question about an effect, so it calls for a quantitative comparison, ideally with a control group or a comparison over time, and with attention to confounding. The second is a question about meaning, so interviews or a phenomenological study would fit, with the families' own accounts as the evidence.

**Kiffer:** And the third question, about who decides how the data are stored and used?

**Sarah:** That's a governance question, and no study design answers it. Under the OCAP principles, the community owns and controls its data, so the answer is settled through a research agreement before any data are collected.

**Kiffer:** Right. And the three answers can sit in one project. That's what the mixed methods designs and Two-Eyed Seeing from the lesson describe, with the community shaping the questions from the start.

**Sarah:** Let's pull it together. My first takeaway is that a good comparison depends on what differs between the groups. Snow's water companies were convincing because the households were mixed street by street, and his result still deserves questions about its denominator.

**Kiffer:** My second takeaway is that honest researchers can produce false findings. Testing ten outcomes with no true effect gives about a forty percent chance of at least one significant result, and a significant result is only as believable as the field's mix of true hypotheses, study size and analytic discipline.

**Sarah:** And third, match the evidence to the question. Trials are strongest for intervention effects, qualitative work and lived experience answer questions about meaning, and frameworks like OCAP answer questions about who controls the data.

**Kiffer:** If you'd like more practice, rework the one thousand hypotheses problem with your own numbers. Try one in fifty hypotheses being true and see what happens to the share of significant findings that are real.

**Sarah:** Next time, it's Lesson two, Systematic Reviews and Meta-Analysis, where we look at how many studies are combined into a single answer, and how the problems from this episode carry into that work.

**Kiffer:** Take care, everyone.

**Sarah:** See you in Lesson two.
