# Lesson 8: Sampling, Selection Processes & External Validity

*Companion-podcast transcript • Sarah & Kiffer*  
*Office Hours episode to listen to after working through the lesson*

---

**Sarah:** Welcome back to Office Hours. I'm Sarah.

**Kiffer:** And I'm Kiffer. This episode goes with Lesson eight, Sampling, Selection Processes and External Validity.

**Sarah:** It's meant for after you've worked through the lesson. We'll bring in some perspective on why selection matters in real studies, some critique of the ways it gets misread, a few of the questions students tend to find thorny, and some extra worked examples, including one that's harder than anything on the lesson page.

**Kiffer:** Along the way there are a few places where we ask you to work something out before we give the answer. You'll hear a few seconds of quiet at those points, and you're welcome to pause the audio if you'd like more time to think it through.

**Sarah:** We have five questions. First, does a study sample have to be representative to give a valid answer? Second, which way does Berkson's bias push an association? Third, does a high response rate mean a survey is free of nonresponse bias? Fourth, how can a real risk factor vanish from prevalence data? And fifth, when does a result from one population carry over to another?

**Kiffer:** Question one. Does a sample have to be representative? I'd like to start here because students often treat selection bias and an unrepresentative sample as the same thing. The lesson defines selection bias more narrowly. It arises when getting into the study, or staying in it, depends jointly on the exposure and the outcome, or on their causes.

**Sarah:** Let's test that with numbers. Here's a made-up cohort of two thousand people. One thousand are exposed, and one hundred of them develop the outcome. One thousand are unexposed, and fifty of them develop the outcome. So the risk is ten percent compared with five percent, a true risk ratio of two.

**Kiffer:** Now suppose exposed people are much harder to recruit. Only thirty percent of them enroll, compared with eighty percent of unexposed people, and enrolment has nothing to do with the outcome. What happens to the risk ratio?

**Sarah:** Thirty percent of the exposed group is three hundred people, with thirty cases, so the risk is still ten percent. Eighty percent of the unexposed group is eight hundred people with forty cases, still five percent. The risk ratio is still two.

**Kiffer:** And the sample is badly unrepresentative. Half the source population is exposed, and only about twenty-seven percent of the sample is, three hundred out of eleven hundred. Any prevalence you report from this sample is wrong, while the association is untouched.

**Sarah:** Then let's make selection depend on both. Say exposed people who develop the outcome enroll at sixty percent, and everyone else enrolls at eighty percent. The exposed group now has sixty cases and seven hundred and twenty non-cases, so the risk is sixty divided by seven hundred and eighty, about 7.7 percent. The unexposed risk is still five percent. The risk ratio drops to about 1.5.

**Kiffer:** That's the distinction in one example. Unrepresentative sampling damages description. Selection that depends on exposure and outcome together damages the association. And if you enrolled ten times as many people through the same process, you'd get the same 1.5 with a narrower confidence interval. That's what it means to call selection bias structural.

**Sarah:** This is where I think we disagree a little. Kenneth Rothman, John Gallacher and Elizabeth Hatch published a paper in the International Journal of Epidemiology in 2013 called "Why representativeness should be avoided." Their argument is that scientific generalization comes from understanding mechanisms and making controlled comparisons, and that representative sampling is mainly for description. I find that hard to accept as a general rule for public health.

**Kiffer:** I'm fairly sympathetic to their argument for causal questions. If I want to know whether an exposure causes a disease, I'd often rather recruit a group that's easy to follow and has a wide spread of exposure, even if it looks nothing like the general population. Their point is that a representative sample averages over subgroups, and that the average can hide the variation you most want to understand.

**Sarah:** But in the same journal that year, Shah Ebrahim and George Davey Smith wrote a commentary that pushes back. They pointed out that volunteer cohorts can produce biased associations when selection is linked to confounders, and that you can't predict the direction of that bias. They concluded that representativeness should be judged in each particular setting. And public health agencies need prevalence figures, which do require a representative sample.

**Kiffer:** I agree with them on prevalence, and on the warning about volunteers. Where I'd hold my position is that a representative sample is neither necessary nor sufficient for a valid causal estimate. Our cohort example showed a very unrepresentative sample with a perfectly correct risk ratio.

**Sarah:** So the practical rule we'd both accept is to name the purpose first. If the study estimates how common something is, demand a representative sample. If it estimates an association, ask whether selection depended on both exposure and outcome, and ask separately whether the effect modifiers in the sample match the population you care about.

**Kiffer:** Question two. Which way does Berkson's bias push? Sarah, I'd like you to try this one. Here's a made-up town of ten thousand adults. Twenty percent have condition A, ten percent have condition B, and the two are completely independent. People with neither condition are admitted to hospital with probability 0.1. People with only A, or only B, are admitted with probability 0.4. People with both are admitted with probability 0.7. Among in-patients, will A and B look positively or negatively associated? Take a few seconds.

*(Pause)*

**Sarah:** My instinct says positive. People with both conditions have the highest admission probability, seventy percent, so the hospital should be packed with people who have both. Two conditions turning up together more often than they should sounds like a positive association.

**Kiffer:** That's the picture many people carry. Let's check it by counting. How many people fall in each group in the town, and how many of each end up in hospital?

**Sarah:** Okay. Both conditions is twenty percent times ten percent of ten thousand, which is two hundred people. A only is eighteen hundred, B only is eight hundred, and neither is seventy-two hundred. Admitted, that's seventy percent of two hundred, so one hundred and forty with both. Forty percent of eighteen hundred is seven hundred and twenty with A only. Forty percent of eight hundred is three hundred and twenty with B only. And ten percent of seventy-two hundred is seven hundred and twenty with neither.

**Kiffer:** Now compare the share of in-patients who have B, among those with A and among those without A.

**Sarah:** Among in-patients with A, it's one hundred and forty out of eight hundred and sixty, about sixteen percent. Among in-patients without A, it's three hundred and twenty out of one thousand and forty, about thirty-one percent. So B is less common among people who have A. That's a negative association, and my instinct was wrong.

**Kiffer:** The odds ratio comes out at about 0.44, from a town where the true odds ratio is one. Why do you think the intuition failed?

**Sarah:** I was looking at the size of the admission probability for the both group. What matters is how much each condition adds. Among people without A, having B raises admission from ten percent to forty percent, four times higher. Among people with A, having B raises it from forty to seventy, less than double. B adds less once you already have a reason to be in hospital.

**Kiffer:** That's the key. When each condition is a separate reason for admission, the second condition adds relatively little, and the association among in-patients turns negative. On a causal diagram, admission is a common effect of A and B, and analysing only in-patients conditions on that common effect, which opens a path between two conditions that have nothing to do with each other. A positive association needs a combination that is admitted far more readily than either condition alone would predict. Either way, the remedy is the same one from the lesson, drawing controls from the community the cases came from.

**Sarah:** The healthy worker effect is the occupational version of a filter like that. The lesson's illustrative asbestos cohort has an all-cause standardized mortality ratio of 0.85, with mesothelioma at 8.2. What's the most useful question to ask about a ratio below one like that?

**Kiffer:** I'd ask who the comparison group is. The general population includes people too ill to work, so it's an unfair yardstick for any workforce. Comparing exposed workers with unexposed workers in the same industry puts both groups through the same hiring filter. The causes of death with a clear link to the exposure, such as mesothelioma, also show the hazard through the filter. There's a second filter that works over time, the healthy worker survivor effect. People who become ill tend to leave the job, so the workers who stay longest are the healthiest, and long exposure can look harmless partly because of who stayed. Lagged analyses and internal comparisons help with that one too, though they can't fully remove it.

**Sarah:** Question three. Does a high response rate mean a survey is free of nonresponse bias? Here's a pair of made-up surveys on smoking. Survey A has an eighty percent response rate. Ten percent of respondents smoke, and thirty percent of the nonrespondents smoke. Survey B has a forty percent response rate. Ten percent of respondents smoke, and twelve percent of nonrespondents smoke. Which survey's smoking estimate is further from the truth? Take a few seconds.

*(Pause)*

**Kiffer:** Both surveys report ten percent, so the question is what the true value is in each. For Survey A, it's eighty percent of people at ten percent plus twenty percent of people at thirty percent. That's eight points plus six points, so fourteen percent. The survey is off by four percentage points.

**Sarah:** And Survey B is forty percent at ten percent plus sixty percent at twelve percent, which is four points plus 7.2 points, so 11.2 percent. It's off by only 1.2 points.

**Kiffer:** So the survey with half the response rate has less than a third of the bias. The bias is the nonresponse rate times the gap between respondents and nonrespondents. In Survey A, that's 0.2 times twenty points, which gives four. In Survey B, it's 0.6 times two points, which gives 1.2.

**Sarah:** That matches what survey methodologists have reported. Groves and colleagues wrote in 2008 that nonresponse rates are poor indicators of nonresponse error, and that the error often varies more across different estimates in the same survey than across surveys with different response rates.

**Kiffer:** Which makes sense from our formula, because the gap between respondents and nonrespondents is different for each variable. A survey can be nearly unbiased for one question and badly biased for another.

**Sarah:** There's a real example of the direction on the lesson page. Keyes and colleagues linked twenty waves of the U.S. National Health Interview Survey, from 1990 to 2009, to death records. Respondents had lower mortality than the general population in every year, with or without survey weights. The rate ratio was about 0.86 for men and 0.89 for women.

**Kiffer:** And part of that, as the lesson notes, is the sampling frame, which leaves out people outside private households. Canada has the same issue. The Canadian Community Health Survey excludes people living on reserves and other Indigenous settlements in the provinces, full-time members of the Canadian Forces, and the institutionalized population. Statistics Canada says these exclusions are less than three percent of adults.

**Sarah:** Three percent sounds small.

**Kiffer:** For many estimates it is small. For questions where the excluded groups differ sharply, such as the health of First Nations communities or of people in long-term care, the survey says little or nothing, however high the response rate. Coverage of the frame and response within it are separate questions, and both deserve a look.

**Sarah:** Attrition works the same way over time. The lesson's cannabis and psychosis reflection describes thirty percent lost to follow-up, and the number that matters is whether the people lost were the heavy users who would have developed symptoms. How would you check that in practice?

**Kiffer:** I'd start by comparing the people who left with the people who stayed on their baseline exposure and risk factors. Then I'd run the two extreme scenarios for the missing outcomes, assuming first that every heavy user who dropped out developed symptoms, and then that none did. If the conclusion holds at both extremes, attrition is unlikely to explain it. If it flips, the honest report says the result depends on who was lost, and weighting methods such as inverse probability of censoring weighting can only help with the part of dropout that measured variables explain.

**Sarah:** Question four. How can a real risk factor vanish from prevalence data?

**Kiffer:** The lesson gives the relationship that prevalence is approximately incidence times duration. Here's a made-up disease. Among unexposed people, incidence is one per thousand person-years and the average duration is ten years. The exposure doubles incidence, and it also halves the average duration, because it makes the disease more often fatal. What would a cross-sectional survey show for this exposure? Take a few seconds.

*(Pause)*

**Sarah:** Unexposed, it's 0.001 per year times ten years, which gives a prevalence of about one percent. Exposed, it's 0.002 per year times five years, which also gives about one percent. So the prevalence ratio is about one, and the survey would suggest the exposure does nothing.

**Kiffer:** And that's true even though the exposure doubles the rate of new disease. The survey answers a question about who is currently living with the disease, and the exposure has removed some of its cases by killing them sooner. That's Neyman bias toward the null. The same arithmetic can even reverse the sign, if the exposure shortens duration by more than it raises incidence, so a cross-sectional study can make a harmful exposure look protective.

**Sarah:** And if an exposure improved survival without changing incidence, it would show up in the prevalence data as a risk factor.

**Kiffer:** Right, a bias away from the null. The remedy is to count new cases, with a cohort or a case-control study that enrolls cases at diagnosis. Survivorship bias is the cohort version. If people enter the cohort some time after diagnosis, the ones who died early never make it into your data, which is why the clock should start at diagnosis or at the start of treatment.

**Sarah:** Question five. When does a result from one population carry over to another? This is our harder example, and it combines transportability with the point from the lesson that effect modification depends on the scale. Here's a made-up stroke prevention trial. Seventy percent of the participants are high risk, with a five-year stroke risk of twenty percent without treatment. Thirty percent are low risk, at five percent. The drug multiplies the risk by 0.75 in both groups, so the risk ratio is 0.75 everywhere.

**Kiffer:** And the target population, a primary care practice, is only twenty percent high risk and eighty percent low risk. The trial reports its average risk reduction. Does it transport to the practice? Take a few seconds.

*(Pause)*

**Sarah:** I'd be tempted to say yes. The risk ratio is the same in both groups, so there's no effect modification, and transport problems come from effect modifiers that are distributed differently.

**Kiffer:** That's the trap I was hoping we'd catch. You've said there's no effect modification, but on which scale?

**Sarah:** Oh. On the ratio scale. On the difference scale it's different. In the high-risk group, a quarter of twenty percent is five percentage points. In the low-risk group, a quarter of five percent is 1.25 points. So baseline risk modifies the risk difference.

**Kiffer:** Good. Now weight those two differences by each population's mix of high-risk and low-risk people.

**Sarah:** In the trial, it's 0.7 times five plus 0.3 times 1.25. That's 3.5 plus 0.375, about 3.9 percentage points, or about thirty-nine strokes prevented per thousand people treated. In the practice, it's 0.2 times five plus 0.8 times 1.25, which is one plus one, so two percentage points, or twenty per thousand.

**Kiffer:** So the benefit in absolute terms is roughly halved. The risk ratio of 0.75 does transport in this example, and the absolute benefit doesn't. What we just did by hand is standardization, averaging the subgroup effects over the target population's mix. Inverse probability of selection weighting reaches the same place by reweighting each trial participant. If the practice is deciding whether the drug is worth its side effects and cost, the absolute number is the one it needs.

**Sarah:** And this all assumes there's no other effect modifier, that the drug is the same version, and that strokes are measured the same way, which are the other conditions on the lesson's list.

**Kiffer:** Right. A larger trial would give a more precise estimate for the trial population, and it would leave the gap between the two populations exactly where it was.

**Sarah:** Let's pull it together. First, ask what a sample is for. An unrepresentative sample damages descriptive estimates such as prevalence, and it biases an association only when selection depends on both exposure and outcome.

**Kiffer:** Second, filters are easy to misjudge by intuition. Berkson's bias can run in either direction, and response rates are poor guides to nonresponse bias, so count it out with the numbers or with a diagram.

**Sarah:** And third, transportability depends on the scale. A ratio that holds steady across groups can still give a very different absolute benefit in a population with a different mix of baseline risk.

**Kiffer:** If you'd like more practice, rework the Berkson hospital example from this episode, and try admission probabilities that make the combination especially likely to be admitted.

**Sarah:** Next time, it's Lesson nine, Information Bias and Data Quality, where the people in the study are the right people and the measurements on them go wrong.

**Kiffer:** Take care, everyone.

**Sarah:** See you in Lesson nine.
