# Lesson 5: Cohort Studies

*Companion-podcast transcript • Sarah & Kiffer*  
*Office Hours episode to listen to after working through the lesson*

---

**Sarah:** Welcome back to Office Hours. I'm Sarah.

**Kiffer:** And I'm Kiffer. This episode goes with Lesson five, Cohort Studies.

**Sarah:** It's best heard once you've worked through the lesson. We'll add some perspective on how cohort studies and trials get used in practice, some critique of where they go wrong, a few of the questions students tend to find thorny, and some extra worked examples, including a couple that are harder than the ones on the lesson page.

**Kiffer:** There are also a few places where we'll ask you to work something out before we give the answer. When that happens, you'll hear a few seconds of quiet. Pause the audio if you'd like more time, and come back when you have an answer of your own.

**Sarah:** Here's the plan. First, what loss to follow-up does to a risk-based cohort, and why the ten percent rule of thumb can mislead. Second, how person-time gets allocated when exposure changes, and a trap called immortal time bias. Third, the healthy-worker effect and the choice of comparison group. Fourth, why a vaccine can look weak in exactly the places where it's working best. And last, a question Kiffer and I see differently, which is whether a well-designed cohort study can answer the questions we usually leave to randomized trials.

**Kiffer:** Question one. A risk-based design counts people, so its denominator is the number of subjects in each exposure group. That only works when everyone is watched for the whole risk period. So what happens to the risk ratio when some people drop out?

**Sarah:** Let's make up a cohort to find out. Imagine eight hundred surgical patients, four hundred who smoke and four hundred who don't, followed for thirty days after surgery for wound infection. Forty of the smokers and ten of the nonsmokers can't be reached at thirty days. Among the three hundred and sixty smokers who were followed, thirty-six got an infection, which is ten percent. Among the three hundred and ninety nonsmokers who were followed, thirteen did, which is about 3.3 percent.

**Kiffer:** So the observed risk ratio is ten percent divided by 3.3 percent, which is three. Smokers have three times the thirty-day risk of infection among the people we could follow. And overall, fifty of eight hundred were lost, which is about six percent, comfortably under the ten percent cut-point that some authors use.

**Sarah:** Which sounds reassuring until you look at the two groups separately. Ten percent of the smokers were lost, and only two and a half percent of the nonsmokers. So here's the question for everyone. What is the lowest the risk ratio could possibly be, if the lost patients had the worst possible outcomes for our conclusion? Take a few seconds.

*(Pause)*

**Kiffer:** The lowest value comes from assuming none of the forty lost smokers got an infection and all ten lost nonsmokers did. Then the smokers have thirty-six infections out of four hundred, which is nine percent. The nonsmokers have thirteen plus ten, or twenty-three, out of four hundred, which is 5.75 percent. Nine divided by 5.75 is about 1.6.

**Sarah:** And the highest goes the other way. All forty lost smokers infected gives seventy-six out of four hundred, or nineteen percent, and no lost nonsmokers infected gives thirteen out of four hundred, or 3.25 percent. Nineteen divided by 3.25 is about 5.8.

**Kiffer:** So the true risk ratio could sit anywhere from about 1.6 to about 5.8. Those are extreme assumptions, and the truth is almost certainly closer to three. Still, the exercise tells you two things. Even the worst case shows smokers at higher risk, so the direction of the finding holds up. And the size of the effect is much less certain than the single number suggests.

**Sarah:** The thing I'd take from it is that the ten percent rule looks at the total, and the bias comes from the difference between groups. Six percent overall can hide a loss that is four times higher in one arm than the other.

**Kiffer:** That's right. When you read a cohort study, calculate the loss in each exposure group separately, and ask whether the people who left are likely to differ in their risk from the people who stayed. A smoker who had a bad wound and went to a different hospital is a very different loss from one who moved away healthy.

**Sarah:** Question two, which is the one I find hardest. In a rate-based design, a person's time can be split between exposure categories. Before they start an exposure, their time counts as unexposed, and afterward it counts as exposed. Kiffer, give me a problem and I'll try it.

**Kiffer:** Here's a made-up one. Four hundred adults with no heart disease are followed for five years. One hundred of them start a new medication at the two-year mark and stay on it. The other three hundred never start it. To keep the arithmetic simple, we'll ignore the small amount of person-time lost when someone becomes a case. In the first two years there are sixteen heart attacks, all among people who never start the drug. In the last three years there are eighteen more among the never-starters and six among the starters.

**Sarah:** Okay. I'll compare people who ever took the drug with people who never did. The starters were followed for five years each, so a hundred times five is five hundred person-years, with six cases. That's twelve per thousand person-years. The never-starters have three hundred times five, or fifteen hundred person-years, with sixteen plus eighteen, which is thirty-four cases. That's about 22.7 per thousand. Twelve divided by 22.7 is about 0.53. So the drug cuts the rate of heart attacks roughly in half.

**Kiffer:** That's a big effect. Before we believe it, how many heart attacks could the starters have had in their first two years?

**Sarah:** None. They had to get to the two-year mark without a heart attack in order to start the drug. If one of them had a heart attack in year one, they'd never have become a starter at all. So those first two years were guaranteed to be event-free, and I credited them to the drug.

**Kiffer:** That guaranteed event-free stretch is called immortal time, and counting it as exposed is immortal time bias. It comes from using information about the future, in this case who would later start the drug, to classify people at the start of follow-up. So how do you fix it?

**Sarah:** I split the starters' time. Their first two years, a hundred people times two years, or two hundred person-years, go to the unexposed group. In fact, all four hundred people were unexposed for the first two years, which gives eight hundred person-years with sixteen cases. Then the never-starters add three hundred times three, or nine hundred person-years, with eighteen cases.

**Kiffer:** Good. So when you add up all the unexposed time and all the unexposed cases, what do you get?

**Sarah:** Eight hundred plus nine hundred is seventeen hundred person-years, with thirty-four cases, which is twenty per thousand. The exposed group is the starters' last three years, a hundred times three, or three hundred person-years, with six cases. That's also twenty per thousand. So the rate ratio is exactly one. The drug does nothing in these numbers.

**Kiffer:** And my invented data were built that way on purpose. A drug with no effect at all looked as if it nearly halved the rate, purely from where the time was put. That's the reason the lesson's rule is so strict: time and cases go to whatever category the person was in at that moment.

**Sarah:** Let's add one more layer. Suppose the drug is thought to need a one-year induction period before it could plausibly affect heart attacks. Where does the starters' first year on the drug go, and how much exposed person-time is left? Take a few seconds.

*(Pause)*

**Kiffer:** The first year after starting goes to the unexposed group, because the exposure can't have acted yet. So the starters contribute a hundred person-years to unexposed for year three, and their exposed time shrinks to a hundred times two, or two hundred person-years. Any heart attacks in that first year would move to the unexposed group as well, or be discarded, which is the other option the lesson gives.

**Sarah:** And discarding them is safer when you aren't sure how long the induction period really is, as long as enough exposed time remains for a precise estimate.

**Kiffer:** Right. Immortal time bias can appear wherever people are grouped by something that happens after follow-up begins. If you take one check from this question, ask when each person's exposed time starts, and whether anything had to go right for them to get there.

**Sarah:** Question three is about the comparison group, and occupational cohorts are where it bites hardest. Kiffer, set it up for us with some numbers.

**Kiffer:** Suppose, with made-up numbers, we follow workers at a smelter and count one hundred and eighty deaths. Using provincial death rates for people of the same ages, we'd have expected two hundred.

**Sarah:** So the standardized mortality ratio is one hundred and eighty divided by two hundred, which is 0.9. The workers died at about ninety percent of the provincial rate. A careless reading would be that working at the smelter is good for your health.

**Kiffer:** And that's the healthy-worker effect. The provincial population includes people too sick to hold any job. People who are hired, and who stay hired, are healthier than average to begin with. So an external comparison with general population rates is tilted in the workers' favour before the exposure has done anything.

**Sarah:** Which is where the internal comparison comes in. You compare workers with workers.

**Kiffer:** Yes. Say the same one hundred and eighty deaths split into sixty among workers in high-exposure jobs over eight thousand person-years, and one hundred and twenty among workers in low-exposure jobs over twenty-four thousand person-years. What are the two rates, and what's the rate ratio? Take a few seconds.

*(Pause)*

**Sarah:** Sixty divided by eight thousand is 7.5 per thousand person-years. One hundred and twenty divided by twenty-four thousand is five per thousand. So the rate ratio is 7.5 divided by five, which is 1.5. Inside the workforce, the high-exposure jobs carry about one and a half times the death rate.

**Kiffer:** So the same deaths tell two different stories. Compared with the province, the workers look protected. Compared with each other, the exposure looks harmful. The internal comparison is generally the more trustworthy one, because both groups passed through the same hiring and staying process.

**Sarah:** Although the internal comparison has its own version of the problem. People who feel unwell in a high-exposure job may move to a lighter job or leave. That's the healthy-worker survivor effect, and it pulls the sick out of the exposed group over time.

**Kiffer:** That's a good point, and it links straight back to question two. If someone moves from a high-exposure job to a low-exposure job because they're getting sick, their later time and their eventual death land in the low-exposure group. The person-time rules are correct, and the bias still gets in, because the exposure change was caused by early illness. Lagging exposure by a few years is one common way to soften that.

**Sarah:** Question four moves to trials. The lesson's cholera example showed a vaccine looking weak in a high-coverage area. I want to work a new version to make sure I understand why. Kiffer, give me the numbers.

**Kiffer:** These are made up. Community A has high vaccine coverage. Its unvaccinated residents have a first-year risk of three per thousand, and its vaccinated residents two per thousand. Community B has low coverage. Its unvaccinated residents have a risk of twelve per thousand, and its vaccinated residents four per thousand.

**Sarah:** The direct effect in A is three minus two, divided by three, which is one third, or about thirty-three percent. In B it's twelve minus four, divided by twelve, which is two thirds, or about sixty-seven percent. Same vaccine, and the direct estimate doubles.

**Kiffer:** Now the indirect effect, which is the one that surprises people. Which two groups does it compare, and what does it come to?

**Sarah:** That compares the unvaccinated in the two communities. Twelve minus three, divided by twelve, is nine twelfths, or seventy-five percent. So an unvaccinated person in community A has three quarters less risk than an unvaccinated person in B, purely from living among vaccinated neighbours.

**Kiffer:** That's the whole story in three numbers, and it's worth slowing down on why it happens. In A, so many people are vaccinated that the infection rarely circulates, so even the unvaccinated are well protected, and the vaccinated have little left to gain over them. The direct comparison is small because the comparison group is already benefiting from the vaccine. The lesson calls this interference: one person's vaccination changes another person's outcome.

**Sarah:** So a trial run only in a high-coverage setting could make a good vaccine look mediocre, and a decision maker might drop it.

**Kiffer:** Right, which is why the trial designs in the lesson need at least two comparable populations with different coverage, kept apart so people don't mix. It's also a useful reminder that randomization protects the comparison inside a trial, and the meaning of the number still depends on the setting where it was measured.

**Sarah:** That leads into our last question, which is where we disagree. Can a well-designed cohort study answer the kind of question we usually leave to a randomized trial? My answer is mostly no, and I'll use a real example. For years, observational studies, including the Nurses' Health Study, suggested that women taking hormone therapy after menopause had a much lower risk of coronary heart disease. The trial report describes a large body of observational work pointing to a forty to fifty percent reduction.

**Kiffer:** And then the Women's Health Initiative trial reported its first results for estrogen plus progestin in 2002, and the picture changed.

**Sarah:** Yes. It randomized 16,608 women aged fifty to seventy-nine to estrogen plus progestin or placebo. The hazard ratio for coronary heart disease was 1.29, so the women on hormones had more heart disease. The trial was stopped early, when the result for invasive breast cancer crossed its stopping boundary. For me that's the clearest case for why we call the trial the gold standard.

**Kiffer:** I agree the trial was the decisive evidence. Where I'd push back is on what the story says about cohorts. In 2008, Miguel Hernán and colleagues reanalysed the Nurses' Health Study data to mimic the trial. They compared women who started hormone therapy with women who didn't start it, and they began follow-up at the time of starting. Earlier analyses had compared current users with never users.

**Sarah:** And what did they find?

**Kiffer:** Over the whole follow-up, a hazard ratio of 0.96, and in the first two years after starting, 1.42. The authors describe those estimates as similar to the trial's. The earlier comparison of current users with never users had given 0.68. Much of the gap came from who was being compared and when the clock started, which is the immortal time issue from question two in another form.

**Sarah:** That's impressive, but I'd add two cautions. First, the reanalysis was done after everyone knew the trial's answer, so it's easier to find the right design in hindsight. Second, even a perfect design can't solve unmeasured confounding. Exchangeability can't be tested in an observational study, and the lesson is clear on that.

**Kiffer:** Both fair. My view is moderate. For many exposures, like smoking, air pollution or a job, a trial is impossible or unethical, so a cohort is the best evidence we'll ever get. I think it's worth designing those cohorts as if they were trials, with clear eligibility, a clear start of follow-up and new users of the exposure. That won't remove unmeasured confounding, though it removes the self-inflicted errors.

**Sarah:** I'm still more skeptical than Kiffer about cohort estimates of treatment effects. But I think we agree on the practical rule. When you read a cohort study of an intervention, ask what trial it is trying to imitate, check who is eligible, when follow-up starts and whether the exposed are new users, and give it less weight when those don't line up.

**Kiffer:** Agreed. And when a cohort and a trial disagree, look at those design differences first, before concluding that one of them is simply wrong. Often the two are answering slightly different questions about slightly different people. Let's pull it together.

**Sarah:** First, look at loss to follow-up in each exposure group separately. A small overall loss can hide a large difference between groups, and simple best-case and worst-case bounds show how far the result could move.

**Kiffer:** Second, person-time belongs to the category a person was in at that moment, after any induction period. Classifying people by something that happens later in follow-up creates immortal time, and it can make a useless treatment look protective.

**Sarah:** And third, every estimate depends on its comparison group. Workers compared with the general population look healthy because of who gets hired, and a vaccine's direct effect looks small where coverage is high, because the unvaccinated are already protected.

**Kiffer:** If you'd like more practice, rework the heart attack and medication problem from this episode with the drug started at the three-year mark and a rate of twenty per thousand person-years throughout, and see how much larger the immortal time bias becomes.

**Sarah:** Next time, it's Lesson six, Ecological and Group-Level Studies, where the unit of analysis becomes the group, and herd immunity comes back as a property of whole populations.

**Kiffer:** Take care, everyone.

**Sarah:** See you in Lesson six.
