# Lesson 11: Confounding & Statistical Inference

*Companion-podcast transcript • Sarah & Kiffer*  
*Office Hours episode to listen to after working through the lesson*

---

**Sarah:** Welcome back to Office Hours. I'm Sarah.

**Kiffer:** And I'm Kiffer. This episode goes with Lesson eleven, Confounding and Statistical Inference.

**Sarah:** It's meant for after you've worked through the lesson. We'll bring in some perspective on how these ideas show up in practice and some critique of the usual methods, spend time on the questions students tend to find thorny, and work some extra examples, a few of them harder than the ones on the lesson page.

**Kiffer:** There are a few places where we'll ask you to work something out before we give the answer. When that happens, you'll hear a few seconds of quiet. Pause the audio if you'd like more time to think it through.

**Sarah:** We have five questions. First, how do you recognize a confounder, and why can't the three classical criteria settle it on their own? Second, when a Simpson's paradox appears, which number should you believe? Third, what does a statistically significant finding from a small study actually tell you? Fourth, how does the reason data are missing change what a complete-case analysis gives you? And fifth, a debate between the two of us about adjusting for income and education when studying racial disparities.

**Kiffer:** Question one. How do you recognize a confounder? Let's start with a made-up example so the mechanics are clear. Imagine two thousand adults followed for ten years. Half of them smoke. Among the smokers, seven hundred drink coffee every day and three hundred don't. Among the nonsmokers, two hundred drink coffee and eight hundred don't. Lung cancer risk over the ten years is ten percent in smokers and one percent in nonsmokers, and in this invented population coffee has no effect at all.

**Sarah:** So smoking is tied to coffee drinking, and smoking causes lung cancer on its own. Here's the question for everyone. What's the crude risk ratio for coffee and lung cancer, ignoring smoking? Work out the risk in each coffee group first. Take a few seconds.

*(Pause)*

**Kiffer:** Among the nine hundred coffee drinkers, seven hundred are smokers with a ten percent risk, which gives seventy cases, and two hundred are nonsmokers with a one percent risk, which gives two cases. That's seventy-two cases out of nine hundred, or eight percent. Among the eleven hundred people who don't drink coffee, three hundred smokers give thirty cases and eight hundred nonsmokers give eight, so thirty-eight out of eleven hundred, which is about 3.5 percent.

**Sarah:** And eight percent divided by about 3.5 percent is a crude risk ratio of about 2.3. Coffee looks like it more than doubles the risk of lung cancer, and within each smoking group the risk ratio is exactly one.

**Kiffer:** Right. The pattern to remember is the same one as in the hormone therapy example on the lesson page. When the stratum-specific estimates agree with each other and both differ from the crude estimate, the stratifying variable is acting as a confounder.

**Sarah:** Here's where I think students get stuck, though. The lesson gives three criteria. Associated with the exposure, a risk factor for the outcome on its own, and off the causal pathway. Smoking meets all three here. Why isn't a checklist enough?

**Kiffer:** Because a variable can pass all three tests and still be the wrong thing to adjust for. A collider can meet the same three criteria. One version, often called M-bias, is a variable measured before the exposure that is a common effect of two hidden causes, one linked to the exposure and one linked to the outcome, and adjusting for it opens a path that wasn't there. And the third criterion, off the causal pathway, can't be checked in the data at all. Whether a variable sits between the exposure and the outcome is a claim about how the world works, so you need subject knowledge, usually drawn as a causal diagram.

**Sarah:** The time-varying case makes that very concrete. In HIV care, the C D four count guides when treatment starts, which makes it look like a confounder, and earlier treatment raises the count, which makes it a mediator too.

**Kiffer:** Exactly, and ordinary regression can't treat one variable as both at once. Adjusting for it blocks part of the effect and can open a collider path, and leaving it out leaves the confounding in place. Marginal structural models get around this by weighting people so that, in the reweighted population, treatment no longer depends on the earlier count.

**Sarah:** And confounding by indication is a version of the same problem that doesn't need anything changing over time. The reason for the prescription predicts the outcome.

**Kiffer:** Yes, and the design fixes in the lesson are worth thinking about in plain terms. An active-comparator design compares two drugs given for the same indication, so both groups were sick enough to need treatment. That shrinks the difference in baseline risk before any statistics happen. Adjustment can only remove confounding by what was measured, and the severity that drove a prescribing decision is often only partly in the record.

**Sarah:** Question two. When a Simpson's paradox appears, which number should you believe? I'd like to work one myself. Here's a hypothetical. A hospital offers a new rehabilitation program after hip fracture. Two hundred patients get the program and two hundred get usual care. Among patients under seventy-five, forty-five of fifty in the program recover full mobility, compared with one hundred and twenty of one hundred and fifty in usual care. Among patients seventy-five and older, seventy-five of one hundred and fifty in the program recover, compared with twenty of fifty in usual care. Before I start, everyone can try the overall recovery rates. Take a few seconds.

*(Pause)*

**Kiffer:** Go ahead, Sarah. Start with the stratum percentages and then build up to the overall picture for each group.

**Sarah:** Under seventy-five, the program is ninety percent and usual care is eighty percent. Seventy-five and older, the program is fifty percent and usual care is forty percent. So the program wins by ten points in both groups. For the overall rate, the program averages ninety and fifty, which is seventy percent, and usual care averages eighty and forty, which is sixty percent. So the program is better overall too, and there's no paradox here.

**Kiffer:** Check that with the counts. How many program patients recovered in total, out of the two hundred?

**Sarah:** Forty-five plus seventy-five is one hundred and twenty, out of two hundred, which is sixty percent. That doesn't match my seventy. Oh, I see what I did. I gave the two age groups equal weight, and in the program three quarters of the patients are older.

**Kiffer:** Right. A simple average of the two percentages is only the overall rate when the groups are the same size. So redo usual care with the counts.

**Sarah:** One hundred and twenty plus twenty is one hundred and forty, out of two hundred, which is seventy percent. So the program is sixty percent overall and usual care is seventy percent. The program looks worse overall even though it's better in both age groups. There's the reversal, and it was hiding behind my averaging.

**Kiffer:** And the average you first computed is close to what standardization does on purpose, giving both groups a common set of weights. That's one reason a standardized comparison and a crude comparison can point different ways.

**Sarah:** So which number do we believe? Here I'd say the age-specific ones, because age affected who got the program and also affects recovery.

**Kiffer:** Agreed, and that reasoning is the part to keep. The stratified estimate is the right one here because age is a confounder. The arithmetic alone can't tell you that. Suppose instead the strata were defined by whether a patient had a fall during rehabilitation, and the program itself reduced falls. Then the strata would be defined by something the program affects, and stratifying would hide part of its benefit. The same table of numbers can call for different answers depending on the causal story behind it.

**Sarah:** Is there a real example people can look up?

**Kiffer:** A well-known one comes from a 1986 study comparing treatments for kidney stones. Open surgery succeeded in ninety-three percent of small stones, compared with eighty-seven percent for a less invasive procedure called percutaneous nephrolithotomy. For large stones it was seventy-three percent compared with sixty-nine percent. Overall, though, open surgery came out at seventy-eight percent and the less invasive procedure at eighty-three percent.

**Sarah:** And the explanation is the same as in my example. Open surgery was used mostly for the large stones, which are harder to treat with any method.

**Kiffer:** That's right. Two hundred and sixty-three of the three hundred and fifty open surgery patients had large stones. Stone size drove both the choice of treatment and the chance of success, which makes it a confounder by indication of a very simple kind.

**Sarah:** Question three. What does a statistically significant result from a small study tell you? I think many students carry around the idea that if p is below 0.05, there's only a five percent chance the finding is false.

**Kiffer:** That's the most common misreading of a p-value. The p-value is calculated assuming the null hypothesis is true. It tells you how surprising the data would be in that world. The chance that a significant finding is a false positive depends on two other things, the power of the study and how many of the hypotheses being tested were true to begin with. Let's work a harder example that puts those together.

**Sarah:** Okay. Here's a made-up research field. Researchers test one thousand hypotheses about possible risk factors. In truth, one hundred of them are real effects and nine hundred are not. Each study uses an alpha of 0.05, and like the small rare-disease studies in the lesson, each has only twenty-five percent power to detect a real effect. Of all the results that come out significant, what share are real effects? Take a few seconds.

*(Pause)*

**Kiffer:** Start with the real effects. Twenty-five percent power means a quarter of the one hundred true effects reach significance, which is twenty-five. Now the nulls. Five percent of the nine hundred null hypotheses come out significant by chance, which is forty-five false positives. So there are seventy significant results in total, and twenty-five of them are real.

**Sarah:** Twenty-five divided by seventy is about thirty-six percent. So almost two thirds of the significant findings in this field are false positives, even though every study used the usual five percent threshold.

**Kiffer:** And if the same studies had eighty percent power, you'd have eighty true positives and the same forty-five false ones, so eighty out of one hundred and twenty-five, which is sixty-four percent. Power changes the share of significant findings you can trust. A widely cited 2005 paper titled Why Most Published Research Findings Are False made this argument in general form.

**Sarah:** And the winner's curse sits on top of that. Even the twenty-five real effects that reached significance tend to be overestimates.

**Kiffer:** Yes, because in a low-powered study a real but modest effect only crosses the threshold when chance pushes the estimate upward. So the significant estimates are the ones that happened to land high. When a later, larger study finds a smaller effect, that's often what you're seeing, and it's a reason to look at the confidence interval as well as the p-value.

**Sarah:** Question four. How does the reason data are missing change what a complete-case analysis gives you? Let me set up a hypothetical. A cohort of one thousand adults is asked at follow-up how many alcoholic drinks they have in a typical week. Six hundred are light drinkers averaging three drinks a week, and four hundred are heavier drinkers averaging twenty. Ninety percent of the light drinkers answer the question, but only half of the heavier drinkers do. And suppose the study has no other information that predicts who is a heavier drinker. Which missing data mechanism is this? Take a few seconds.

*(Pause)*

**Kiffer:** It's missing not at random. Whether the answer is missing depends on the answer itself, how much the person drinks, and nothing observed in the study accounts for it.

**Sarah:** So let's see how big the bias is. The true average is six hundred times three plus four hundred times twenty, which is nine thousand eight hundred drinks, divided by one thousand people, so 9.8 drinks a week. Among responders, there are five hundred and forty light drinkers and two hundred heavier drinkers. That's one thousand six hundred and twenty plus four thousand drinks, or five thousand six hundred and twenty, divided by seven hundred and forty people, which is about 7.6.

**Kiffer:** So the complete-case average understates drinking by more than two drinks a week, and nothing in the responders' data would warn you. Now change one detail. Suppose the study measured drinking at baseline, and baseline drinking predicts who skips the follow-up question. Then, within levels of baseline drinking, missingness might no longer depend on the missing values.

**Sarah:** And that would make it missing at random, conditional on an observed variable, so multiple imputation using baseline drinking could recover a valid estimate.

**Kiffer:** That's the practical point. Whether data are missing at random depends on what else you measured. The same pattern of nonresponse can move from the hardest category to a manageable one when a good predictor of missingness has been recorded, which is a strong argument for collecting those predictors at the design stage. And because no test can prove data are missing at random, a sensitivity analysis under different assumptions is still worth reporting.

**Sarah:** Question five is where Kiffer and I disagree. The lesson discusses adjusting for income and education when studying racial disparities, and the example is maternal mortality in the United States. A CDC report covering 2007 to 2016 found that the pregnancy-related mortality ratio for Black women was 3.2 times that of White women. My view is that adjusting for income and education in that kind of analysis usually does more harm than good. Those variables are pathways through which racism acts, so adjusting for them removes part of the disparity and labels the remainder as the effect of race.

**Kiffer:** I agree with the diagnosis, and I'd defend the adjusted analysis when it's framed correctly. If a health authority wants to know how much of a disparity runs through income, that's a fair question, and an analysis with income in the model can help answer it. The error I see more often is in the reading, where a smaller adjusted coefficient is described as the disparity being explained away.

**Sarah:** But even framed correctly, the decomposition assumes you could change income while leaving everything else about a person's position the same. That's Bauer's point about additivity. And the data in that same report cast doubt on the "explained by socioeconomic factors" story. Among women with a college degree or more, the ratio for Black women was 5.2 times the ratio for White women, which is larger than the overall gap.

**Kiffer:** That's a strong piece of evidence, and it supports your side. It shows the disparity can't be read as mainly a matter of education. I'd still keep the adjusted analysis as one tool among several, for planning where interventions might act, and pair it with stratified or intersectional analyses that report rates within groups. My position is that the problem lies in reporting the adjusted coefficient alone, and in calling it the effect of race.

**Sarah:** I'm more skeptical than Kiffer that the adjusted figure helps much, but we agree on a practical rule. State the causal question before choosing covariates, report the unadjusted disparity alongside any adjusted estimate, and describe variables such as income and education as possible pathways when the theory says that's what they are.

**Kiffer:** Let's pull it together, with three things to take away from this episode.

**Sarah:** First, whether a variable is a confounder is a causal judgement. The three criteria help, but a collider or a mediator can look the same in the data, so the decision rests on a causal diagram and subject knowledge.

**Kiffer:** Second, when crude and stratified results disagree, the stratified result is the one to read when the stratifying variable is a confounder, and when you combine strata, weight them by their sizes. The hip fracture example showed how an unweighted average can hide a reversal.

**Sarah:** And third, a significant result from a low-powered study deserves caution twice over. It's more likely to be a false positive than the threshold suggests, and if it's real, its size is probably overstated.

**Kiffer:** If you'd like more practice, rework the field of one thousand hypotheses from this episode with two hundred real effects and fifty percent power, and see how the share of trustworthy findings changes.

**Sarah:** Next time, it's Lesson twelve, Integrated Appraisal of Epidemiological Research, the last lesson of the course, where all of these threats come together in the appraisal of whole studies.

**Kiffer:** Take care, everyone.

**Sarah:** See you in Lesson twelve.
