# Lesson 10: Design-Specific & Temporal Biases

*Companion-podcast transcript • Sarah & Kiffer*  
*Office Hours episode to listen to after working through the lesson*

---

**Sarah:** Welcome back to Office Hours. I'm Sarah.

**Kiffer:** And I'm Kiffer. This episode goes with Lesson ten, Design-Specific and Temporal Biases.

**Sarah:** It's meant for after you've worked through the lesson. We'll add some perspective on how these biases turn up in real research, offer some critique, work through the questions students tend to find thorny with this material, and do some extra worked examples, including a few that are harder than the ones on the lesson page.

**Kiffer:** There are also a few places where we ask you to work something out before we give the answer. When that happens, you'll hear a few seconds of quiet. Pause the audio if you'd like more time, and pick it up again when you're ready.

**Sarah:** We'll take four questions. First, how a trial with perfect randomization can still give a biased answer, and which way each bias pushes. Second, how a drug that does nothing at all can look protective because of immortal time. Third, why five-year survival can climb after a screening program starts while the number of deaths stays the same. And fourth, two problems of time in case-control and trend data, which are time-window bias and the age, period and cohort puzzle.

**Kiffer:** Question one. If randomization worked, why should we still worry about bias inside a trial? Sarah, where do you think students most often go wrong with this?

**Sarah:** Many people treat randomization as a seal of quality on the whole study. Randomization balances the groups at the moment people are randomized, on the things we measured and the things we didn't. Everything after that moment is still open to bias, including who gets enrolled, who knows their group, who actually takes the pills, and how the outcomes get recorded.

**Kiffer:** And the two safeguards students mix up most are allocation concealment and blinding. Concealment protects the moment of enrolment. The person recruiting can't know which group the next person will go to, so they can't steer a sicker or healthier patient into one arm. Blinding protects everything afterwards. It keeps participants, clinicians and outcome assessors from knowing who got what. A trial can have one without the other. A surgical trial can conceal allocation perfectly and still have no way to blind the surgeon.

**Sarah:** Schulz and colleagues, in 1995, looked at 250 trials from thirty-three meta-analyses. Trials with inadequate concealment gave odds ratios exaggerated by about forty-one percent on average, and trials that weren't double-blinded gave odds ratios exaggerated by about seventeen percent.

**Kiffer:** Those are averages across many fields, so treat them as a rough guide for any single trial. The direction is the more useful lesson. Concealment failures and lack of blinding usually push the estimate away from the null, so the treatment looks better than it is. Contamination pushes the other way. Here's a hypothetical community trial of a falls-prevention exercise program for older adults, with made-up numbers. Without the program, twenty percent of people have a fall-related injury over two years. With it, twelve percent do. So the true risk ratio is twelve divided by twenty, which is 0.6.

**Sarah:** Now suppose word of the program spreads, and thirty percent of people in the control communities start doing the same exercises on their own, with the same benefit. What risk ratio would the trial actually observe? Take a few seconds.

*(Pause)*

**Kiffer:** The intervention arm stays at twelve percent. The control arm is now a mix. Seventy percent of the controls still have a twenty percent risk, and thirty percent have a twelve percent risk. Seventy percent of twenty is fourteen, and thirty percent of twelve is 3.6, so the control risk is 17.6 percent. Twelve divided by 17.6 is about 0.68.

**Sarah:** So a forty percent reduction in risk shows up as a reduction of about thirty-two percent, even though the randomization itself was fine. That's the same problem as the smoking cessation trial in the lesson, where national anti-smoking campaigns also reached the control communities.

**Kiffer:** Right, the contrast between the arms shrinks, which is why contamination biases toward the null. The Hawthorne effect works through a different route, and there's a good Canadian study of it. Srigley and colleagues, at the University Health Network in Toronto, used electronic monitoring of soap and hand rub dispensers on two units of an academic hospital for eight months. At dispensers within sight of an auditor, there were 3.75 hand hygiene events per dispenser per hour. At dispensers out of the auditor's sight at the same time, there were 1.48.

**Sarah:** That's roughly three times as many, and the electronic system recorded behaviour whether or not anyone was watching. So in a trial, I'd guess being watched mostly squeezes the difference between the arms.

**Kiffer:** If it raises the behaviour equally in both arms, yes, it compresses the difference toward the null. If one arm is watched more closely than the other, for example because the intervention arm has more study visits, it can push the estimate in either direction.

**Sarah:** That brings me to the place where Kiffer and I disagree, which is intention-to-treat analysis compared with per-protocol analysis. The Coronary Drug Project result from the lesson is the standard warning. Placebo adherers had about fifteen percent five-year mortality, and placebo non-adherers had about twenty-eight percent. I accept that a naive per-protocol comparison is misleading. Still, a patient deciding whether to take a drug wants to know what happens if they actually take it, and intention-to-treat answers a different question, which is what happens when the drug is offered.

**Kiffer:** That's a fair point, and it's the question patients ask. My worry is about how that answer gets estimated. Once you look only at the people who stayed on treatment, the groups are no longer the groups that randomization made. People who stop a treatment often stop for reasons tied to their health. At that point you're back to observational data inside a trial.

**Sarah:** Methodologists have taken that problem seriously. Miguel Hernán, who appears in the lesson's glossary, and James Robins wrote about per-protocol analyses of pragmatic trials in the New England Journal of Medicine in 2017. Hernán's position is that a valid per-protocol estimate needs adjustment for confounding, including factors that change after randomization, and that newer methods make this possible when certain conditions hold. He also points out that for safety outcomes and non-inferiority trials, the per-protocol effect is the one of interest.

**Kiffer:** I'd agree with that, with one caution. Those adjustments are only as good as the data on why people stopped or switched, and that data is often thin. My own view is that intention-to-treat should stay the primary analysis, because it rests on randomization alone. I'd read an adjusted per-protocol estimate as a useful second answer that carries extra assumptions.

**Sarah:** I'd give the per-protocol question more weight than Kiffer does, especially for side effects. We agree on a practical rule, though. Treat intention-to-treat as the primary result. When a paper reports a per-protocol effect, check whether it adjusted for the reasons people stopped or switched, and set aside any simple comparison of adherers with all controls.

**Kiffer:** Question two. How can a drug that does nothing look protective in an observational study? This is immortal time bias, and Sarah, I'd like you to work through a hypothetical cohort for us.

**Sarah:** Okay. Here are the made-up data. One thousand people are followed for one year after leaving hospital with a heart attack. Four hundred of them fill a prescription for a drug, each one at three months. The drug has no effect at all. Among the four hundred users there are thirty deaths, all after the prescription, because you have to be alive to fill it. Together the users contribute four hundred person-years. The six hundred non-users contribute five hundred and fifty person-years, with sixty-five deaths.

**Kiffer:** Before Sarah goes on, try the naive analysis yourself. Using the groups as they're defined, ever-users and never-users, what's the death rate in each group, and what's the rate ratio? Take a few seconds.

*(Pause)*

**Sarah:** Users have thirty deaths over four hundred person-years, which is 0.075 per person-year. Non-users have sixty-five deaths over five hundred and fifty person-years, which is about 0.118. The rate ratio is 0.075 divided by 0.118, which is about 0.63. So the drug seems to cut deaths by more than a third.

**Kiffer:** And we built this world so that the drug does nothing. Where does the problem come from in these particular numbers?

**Sarah:** From the first three months for each user. Four hundred people times a quarter of a year is a hundred person-years in which no user could die. So I'll take those hundred person-years out of the users' denominator. That leaves thirty deaths over three hundred person-years, which is 0.10. Non-users stay at about 0.118. The rate ratio is about 0.85. It's still protective.

**Kiffer:** That's a very common place to stop. You've taken the immortal time away from the users, which is the right first step. Where did those hundred person-years go?

**Sarah:** Nowhere. I dropped them. Those people were alive and off the drug for three months, though, so that time is unexposed time. It belongs with the non-users. That makes the unexposed denominator six hundred and fifty person-years, with the same sixty-five deaths, so the rate is 0.10. The exposed rate is also 0.10, and the rate ratio is one.

**Kiffer:** Which is the truth we built in. The lesson describes two errors, crediting the immortal time to the exposed group or excluding it, and your first fix was the second error. Excluding the time leaves the unexposed rate too high, because you've thrown away a hundred person-years in which nobody died. Time-dependent exposure classification puts each stretch of a person's time in the group they were actually in at that moment.

**Sarah:** And the Oscar example in the lesson is the same error outside medicine, isn't it?

**Kiffer:** It is, and it has a Canadian connection. A 2001 study reported that Academy Award winners lived about 3.9 years longer than their peers. A McGill team, Sylvestre, Huszti and Hanley, reanalysed it in 2006, counting winners as winners only from the year they won. The advantage shrank to about one year, within the margin of error. Samy Suissa, whose work on immortal time runs all through this lesson, is at McGill as well.

**Sarah:** So what's the quick check you'd run on any observational drug study?

**Kiffer:** I'd ask, at the moment follow-up starts, does everyone already belong to a group? If being in the exposed group depends on something that happens later, there's immortal time to look for.

**Sarah:** Question three. Why can five-year survival rise sharply after a screening program begins while the number of deaths stays the same? I'd like to do a harder example that combines lead time with overdiagnosis.

**Kiffer:** Go ahead. Use made-up numbers, and make the screening completely useless, so that we know any improvement we see is an artefact of how survival is measured.

**Sarah:** Before screening, a hypothetical region diagnoses one thousand cases of a cancer each year from symptoms. Of those, three hundred die within two years of diagnosis, two hundred more die between two and five years, and five hundred are alive at five years. So five-year survival is fifty percent. Then screening starts. It finds the same thousand cancers, each one three years before symptoms would have appeared. It also finds six hundred small tumours that would never have caused symptoms or death.

**Kiffer:** Here's the question. Nobody's date of death changes at all. What's the new five-year survival among the screen-detected cancers? Take a few seconds.

*(Pause)*

**Sarah:** The three hundred who used to die within two years of symptoms now die within five years of their screening diagnosis, because two plus three is five. They still count as deaths. The two hundred who died between two and five years now die between five and eight years after their screening diagnosis, so they count as five-year survivors. The five hundred who survived before still survive. That's seven hundred survivors among the thousand. Then add the six hundred overdiagnosed cancers, who all survive. That gives thirteen hundred survivors among sixteen hundred cases, which is about eighty-one percent.

**Kiffer:** So survival goes from fifty to about eighty-one percent, with exactly the same deaths. Notice the second signature as well. The number of diagnosed cases rose from one thousand to sixteen hundred a year, a sixty percent rise in incidence, with no change in mortality. Rising incidence with flat mortality is the pattern that points to overdiagnosis.

**Sarah:** The PSA example in the lesson fits this pattern, with five-year prostate cancer survival in the United States rising from about seventy-five percent to over ninety-nine percent while mortality fell only modestly. Is there a Canadian example of the same pattern that students could look up?

**Kiffer:** There is. In Quebec, about four hundred and forty thousand babies were screened between 1989 and 1994 with urine tests for neuroblastoma, a cancer of nerve cells that affects young children. The screening found signs of the cancer about twice as often as expected, and the neuroblastoma death rate afterwards was essentially the same as in unscreened children elsewhere in North America. The results appeared in the New England Journal of Medicine in 2002. Many of the tumours found by screening would have disappeared on their own, which makes it a very clear case of overdiagnosis.

**Sarah:** Which leaves the hard question of what evidence is good enough. Survival from diagnosis is misleading, so we look at mortality. Even a mortality comparison can mislead, though, if the people who chose screening differ from the people who didn't.

**Kiffer:** Right. People who choose screening tend to be healthier, which is the healthy adherer effect turning up again. That's why randomized screening trials with mortality endpoints are the strongest evidence. In my view, when a screening claim rests only on survival statistics, it's reasonable to treat the benefit as unproven until someone shows the mortality data.

**Sarah:** Question four is about two problems with time that look quite different from each other. The first is time-window bias in case-control studies. Kiffer, could you set up an example where the drug has no effect at all, so that any association has to come from the design?

**Kiffer:** Here's a hypothetical case-control study. Cases have, on average, two years of pharmacy records before their index date, because many of them died soon after the event. Controls have four years. Everyone, cases and controls alike, has a twenty percent chance each year of filling a prescription for the drug, and exposure is defined as any fill in the available records. What odds ratio would that study report, even though the drug does nothing? Take a few seconds.

*(Pause)*

**Sarah:** For cases, the chance of never filling in two years is 0.8 times 0.8, which is 0.64, so thirty-six percent are exposed. For controls, the chance of never filling in four years is 0.8 to the fourth power, which is about 0.41, so about fifty-nine percent are exposed. The odds of exposure for cases are 0.36 divided by 0.64, about 0.56. For controls, they're 0.59 divided by 0.41, about 1.44. The odds ratio is 0.56 divided by 1.44, which is about 0.39.

**Kiffer:** That looks like a sixty percent reduction from a drug that does nothing, and it comes entirely from giving controls twice as long to be counted as exposed. The fix is to give everyone the same window, for example the same three hundred and sixty-five days before the index date. The statin and lung cancer example in the lesson shows the real version, where an apparent protective effect disappeared once time was handled properly.

**Sarah:** The second problem is the age, period and cohort puzzle. I understand the identity, that birth year equals calendar year minus age. What I find harder is seeing why that identity actually stops a model from separating the three.

**Kiffer:** Here's a made-up illustration. Imagine a world where only the calendar year matters. Every year, the rate of some condition rises by one case per thousand people, at every age at once. That's a pure period effect. Now imagine a second world, where each later birth year adds one case per thousand, and each extra year of age also adds one case per thousand. Could you tell those two worlds apart from any table of rates?

**Sarah:** Let me try. In the second world, take two people in the same survey, one a year older than the other. The older one gains one from age but was born a year earlier, so loses one from birth year. They come out the same. So within any survey, every age has the same rate. And the same age in the next survey was born a year later, so the rate is one higher. That's exactly the first world.

**Kiffer:** Exactly. Since birth year plus age always gives the calendar year, one story can be rewritten as the other, and no amount of extra data can tell them apart. That's why separating them takes theory or outside information. For smoking and lung cancer, the outside information is strong. We know when each generation took up smoking, and we know lung cancer follows smoking by decades, so a cohort explanation for rising rates in women has support from outside the trend data.

**Sarah:** Here's my sticking point, though. If the choice of constraint can change the answer, isn't every age, period and cohort analysis an assumption dressed up as a result?

**Kiffer:** That's a fair worry, and the constraint really does matter. A good analysis states its constraint, explains why it's plausible from outside evidence, and shows whether the conclusion holds under other reasonable choices. If the conclusion flips when the constraint changes, the data alone can't settle the question, and the paper should say so.

**Sarah:** Let's pull it together. First, randomization protects a trial only at the moment of randomization itself. Concealment, blinding, adherence and contamination decide what happens after that, and they push the estimate in different directions.

**Kiffer:** Second, for any observational drug study, ask whether everyone belongs to a group at the moment follow-up starts. If exposure depends on surviving long enough to receive it, put the time before exposure in the unexposed group and keep it in the analysis.

**Sarah:** And third, in screening and trend data, check which clock the numbers run on. Survival from diagnosis moves with the diagnosis date, case-control exposure depends on how long the records are, and age, period and cohort can't be separated without an assumption that someone has to state.

**Kiffer:** For more practice, rework the time-window example from this episode with a ten percent chance of filling each year, and see how far the false odds ratio moves toward one.

**Sarah:** Next time, it's Lesson eleven, Confounding and Statistical Inference, which finishes the bias material and brings random error and systematic error back into focus.

**Kiffer:** Take care, everyone.

**Sarah:** See you in Lesson eleven.
