Sampling, Selection Processes &
External Validity
Evaluating Epidemiological Research
Learning objectives for this lesson:
- Define selection bias as a structural problem in how data are generated, distinct from confounding and from a lack of representativeness
- Explain Berkson’s bias and identify when hospital-based sampling distorts exposure-outcome associations
- Describe the healthy worker effect, interpret standardized mortality ratios in occupational epidemiology, and identify design strategies that reduce enrollment-stage selection bias
- Distinguish attrition bias from nonresponse bias and evaluate strategies for mitigating each
- Recognize prevalence-incidence (Neyman) bias and explain how cross-sectional designs miss rapidly fatal cases
- Identify survivorship bias in cohort studies and its implications for causal inference
- Evaluate transportability of study findings to target populations with differing characteristics
- Critically assess whether epidemiological studies have adequately addressed selection-related threats to validity
This course was developed by Dr. Kiffer G. Card.
Glossary: Key Terms, People & Concepts
📚 Reference page, available throughout the lesson
This glossary collects the key concepts, people, and ideas you will meet in this lesson. Use it as a reference while you work through the material, or as a review before assessments. Type in the search box to filter entries.
Selection Bias Mechanisms
Introduction and Overview
An earlier lesson worked on what we measure and how we model causation. This lesson turns to the third great source of bias in observational research: who ends up in the study at all. Even with perfect measurement and a correctly specified DAG, a study can produce a wrong answer because of how participants were chosen at the outset, because of who dropped out during follow-up, or because the people available to study are themselves a survival-filtered subgroup. Across three content sections we work through the canonical mechanisms in order: this section covers selection biases that arise at enrollment (Berkson's bias and the healthy worker effect); a later section covers attrition during follow-up and nonresponse at data collection; a later section covers prevalence–incidence (Neyman) bias, survivorship bias, and the related question of transportability, whether a study's findings hold in a different population. By the end of the lesson, you should have a working vocabulary for “the sample is wrong” that complements the “the variables are wrong” vocabulary from an earlier lesson, ready to be combined with the information-bias inventory of a later lesson. The unifying structural account, selection bias as conditioning on a common effect of exposure and outcome, comes from Hernán, Hernández-Díaz, & Robins (2004).
Learning Objectives
- 1.1 Define selection bias as a structural problem in how data are generated, distinct from confounding.
- 2.1 Explain the mechanism of Berkson’s bias and recognize when hospital-based case-control designs produce spurious associations.
- 3.1 Describe the healthy worker effect and interpret SMRs in occupational cohort studies.
- 3.2 Identify design strategies (population-based controls, internal comparisons, lagged analyses, cause-specific outcomes) that mitigate enrollment-stage selection bias.
1.1 What Is Selection Bias?
Selection bias arises when entry into a study, or retention in it, depends jointly on the exposure and the outcome, or on their causes. On a causal diagram, selection is then a common effect of exposure and outcome, and analysing only the people selected conditions on it (Hernán, Hernández-Díaz, & Robins, 2004). Unlike confounding, which involves a common cause of exposure and outcome, selection bias distorts the exposure-outcome relationship through the very mechanics of who ends up being studied.
The key concept below restates the definition in compact form and adds, following Greenland (2003), that selection bias is a structural problem in how data are generated, which larger sample sizes cannot fix.
Key Concept: Selection Bias
Selection bias arises when entry into the study, or retention in it, depends jointly on the exposure and the outcome (or on their causes). It is a structural problem in how data are generated, not a statistical problem that can be fixed by larger sample sizes (Greenland, 2003).
The definition separates two kinds of estimate. When selection depends on the exposure alone or on the outcome alone, descriptive estimates from the sample, such as the prevalence of the exposure or of the outcome, can be biased, while the exposure-outcome odds ratio remains unbiased; the case-control design, which samples on the outcome, relies on this property. The odds ratio is biased when selection depends on both. Whether the participants resemble the target population is a separate question of external validity, which Section 3 (7.1 Transportability of Study Results) takes up.
In this section, we examine two classic mechanisms of selection bias: Berkson’s bias and the healthy worker effect. Both illustrate how the process of selecting participants into a study can create spurious associations or mask real ones.
2.1 Berkson’s Bias
Berkson's bias is the first of the two enrollment mechanisms named above, and it arises in hospital-based case-control studies, where both cases and controls are drawn from a hospital population. The case study below opens the topic with Berkson’s own illustration: diabetes mellitus and cholecystitis in hospital in-patient data.
In 1946, Joseph Berkson used diabetes mellitus and cholecystitis (gallbladder inflammation) to show how hospital data can mislead. In a sample drawn from hospital in-patients, the two conditions can appear associated whatever their relationship in the general population, so an association seen among in-patients could tempt researchers to propose a biological link that does not exist. What goes wrong?
The answer lies in Berkson’s bias, first described by Berkson (1946). This form of selection bias occurs specifically in hospital-based case-control studies. The core insight is that people with two conditions are more likely to be hospitalized than people with only one condition. When you draw both cases and controls from a hospital population, you artificially inflate the co-occurrence of conditions.
The mechanism box below sets it out with two conditions, A and B, that are independent in the general population, and shows why a person with both conditions has a higher probability of hospitalization than a person with either one alone.
The Mechanism of Berkson’s Bias
Consider two conditions, A and B, that are independent in the general population. A person with condition A has some probability of hospitalization (pA), and a person with condition B has probability pB. A person with both conditions has a hospitalization probability of approximately pA + pB − pA × pB, which is always greater than either alone. Because admission depends on both conditions, restricting the sample to in-patients conditions on a common effect of the two, which can create a spurious association between them within the hospital sample even when none exists in the population. Its direction depends on how the conditions, alone and together, drive admission: when each condition is simply a separate reason for admission, as in this formula, the association among in-patients tends to be negative, and when the combination makes admission especially likely, it can be positive.
Click each card to explore key aspects of Berkson’s bias.
Berkson's bias is the textbook hospital-sampling problem. The next mechanism is its occupational cousin: selection that happens not at the moment of recruitment but at the moment people enter (or stay in) the labour force.
3.1 The Healthy Worker Effect
The healthy worker effect is the second enrollment mechanism named at the start of this section, and it arises in occupational cohort studies that compare workers with the general population. The case study below presents the paradox that it explains, using an illustrative cohort of asbestos-exposed workers whose all-cause mortality is lower than that of the general population despite a known hazardous exposure.
Consider an occupational cohort of asbestos-exposed workers compared with the general population, in which the overall mortality rate among the workers is lower than in the general population (an all-cause standardized mortality ratio, or SMR, below 1.0) despite a known hazardous exposure. All-cause SMRs below 1.0 are a common finding in occupational cohorts. How could workers exposed to a known carcinogen have lower mortality?
This paradox is explained by the healthy worker effect (McMichael, 1976), a form of selection bias inherent to occupational cohort studies. Workers are a selected subgroup of the population: they must be healthy enough to obtain and maintain employment. People who are chronically ill, disabled, or otherwise frail are less likely to enter the workforce.
The three tabs that follow describe how the healthy worker effect operates, how it shapes the interpretation of the standardized mortality ratio (SMR), and the study design strategies that reduce it.
How it works: The general population includes people who are too ill to work, institutionalized, or otherwise selected out of the labor force. When we compare workers to this general population, we are comparing a “healthier” group to a more heterogeneous one. The result: overall mortality appears lower among workers, masking the true hazard of occupational exposures.
Importantly, the healthy worker effect is strongest for causes of death unrelated to the occupational exposure (such as cardiovascular disease) and weakens or reverses for exposure-specific outcomes (such as mesothelioma in asbestos workers).
The standardized mortality ratio (SMR) compares observed deaths in a worker cohort to expected deaths based on general population rates. As a reading guide, an SMR of 0.85 means the cohort had about 15% fewer deaths than those rates predict, while 1.45 means about 45% more. An SMR < 1.0 does not mean the workplace is safe; it reflects the selection of healthier individuals into the workforce.
Table 8.1 illustrates the pattern across four causes of death among asbestos workers, from SMRs below 1.0 for all causes and cardiovascular disease to SMRs well above 1.0 for lung cancer and mesothelioma.
Table 8.1. Illustrative standardized mortality ratios among asbestos workers by cause of death, with the interpretation of each.
| Cause of Death | SMR Among Asbestos Workers | Interpretation |
|---|---|---|
| All causes | 0.85 | Healthy worker effect masks overall risk |
| Cardiovascular disease | 0.78 | Strong healthy worker effect for non-occupational causes |
| Lung cancer | 1.45 | Elevated despite healthy worker effect; true risk is likely higher |
| Mesothelioma | 8.20 | Very strong exposure-specific signal overcomes selection bias |
Note: These values are illustrative based on patterns observed in occupational asbestos studies.
Researchers have developed several strategies to mitigate the healthy worker effect:
- Internal comparisons: Compare exposed workers to unexposed workers within the same workforce, rather than to the general population
- Healthy worker survivor effect adjustment: Account for the fact that workers who remain employed over time are progressively healthier (those who become ill leave the workforce)
- Cause-specific analyses: Focus on outcomes with known biological links to the exposure, where the effect of occupational hazard exceeds the healthy worker selection
- Lagged analyses: Introduce exposure lag periods to account for latency between exposure and disease onset
3.2 Design Strategies Against Enrollment-Stage Bias
The remedies for both enrollment mechanisms follow from their structure. Berkson's bias is avoided by drawing controls from the general population rather than from the hospital, so that hospitalization no longer acts as the selection filter. The healthy worker effect is reduced by the strategies in the Study Design Solutions tab above: internal comparisons of exposed and unexposed workers within the same workforce, adjustment for the healthy worker survivor effect, cause-specific analyses of outcomes with known biological links to the exposure, and lagged analyses that allow for latency. Each strategy works by removing or weakening the dependence of selection on both exposure and outcome, not by enrolling more people through the same process.
Both mechanisms in this section can be reproduced in a simulated population. The hands-on box below sets out a task with the simulator in Interactive 8.1, which holds a true population fixed while the reader sets the participation probability for each combination of exposure and outcome; its presets reproduce Berkson's bias, the healthy worker effect, volunteer self-selection and loss to follow-up.
Hands-on: Selection & Recruitment Bias Simulator
What you'll do: the simulator in Interactive 8.1 holds a true population fixed and lets you set the participation probability for each combination of exposure and outcome. The presets reproduce the classic mechanisms: Berkson's bias, healthy worker effect, volunteer self-selection, and loss to follow-up. What to take away: the same true population can produce a 2×2 table whose odds ratio is double, half, or even reversed compared with the truth, depending on who chooses to enroll. You are not adjusting any analysis; you are watching the bias appear in the data themselves. Try the “Berkson's bias” preset first to reproduce the diabetes–cholecystitis story you just read.
🎯 Interactive 8.1: Selection & Recruitment Bias Simulator
A true population of 2,000 people with a known exposure (E) and outcome (Y). You set the participation probability for each subgroup. When participation depends on both E and Y, the observed exposure–outcome association drifts away from the truth. That is selection bias.
Population (true) vs. Observed sample
Each tile = a person. Bright = enrolled in study; faded = excluded. Color encodes E/Y combination.
2×2 tables & effect estimates
True (population)
| Y+ | Y− | |
|---|---|---|
| E+ | – | – |
| E− | – | – |
Observed (sample)
| Y+ | Y− | |
|---|---|---|
| E+ | – | – |
| E− | – | – |
This section has shown how selection at enrollment, through hospital admission in Berkson's bias and through entry into and retention in the workforce in the healthy worker effect, can create spurious associations or mask real ones. The key takeaways and knowledge check that follow close the section, and the next section turns to selection that operates after enrollment, through attrition and nonresponse.
Key Takeaways
- 1.1 Selection bias arises when entry into a study, or retention in it, depends jointly on the exposure and the outcome, so that analysing only the people selected conditions on a common effect of both. It is structural: it arises from how participants are selected, not from measurement or confounding, and larger samples drawn by the same process cannot fix it.
- 2.1 Berkson’s bias creates spurious associations in hospital-based case-control studies because hospitalization probability depends on having multiple conditions. Two conditions that are independent in the population can appear associated in the hospital sample, as Berkson illustrated in 1946 with diabetes and cholecystitis among hospital in-patients.
- 3.1 The healthy worker effect masks true occupational risks because workers are systematically healthier than the general population. An all-cause SMR below 1.0 does not mean the workplace is safe; the effect is strongest for causes unrelated to the exposure and is overcome by exposure-specific outcomes such as mesothelioma.
- 3.2 Population-based designs and internal comparisons are primary strategies for avoiding these biases, supported by cause-specific and lagged analyses in occupational cohorts. Each works by breaking the joint dependence of selection on exposure and outcome.
1. In Berkson’s bias, the spurious association between two conditions arises because:
2. An occupational cohort study of chemical plant workers reports an SMR of 0.82 for all-cause mortality. The most likely explanation is:
3. Which study design best avoids Berkson’s bias?
Attrition & Nonresponse Bias
Introduction and Overview
An earlier section covered selection bias at the moment of enrollment. The mechanisms in this section operate later: attrition reshapes the cohort over time as participants drop out, and nonresponse hollows out a survey before any follow-up has even started. Both are forms of selection bias, but they are easy to miss because the original recruitment was sound. The key diagnostic question for both is the same: does the dropout (or non-participation) depend on both the exposure and the outcome?
Learning Objectives
- 4.1 Recognize when loss to follow-up after enrollment is differential with respect to exposure and outcome, and explain why that, not the absolute attrition rate, is what matters.
- 4.2 Evaluate strategies for detecting and adjusting for attrition (baseline comparisons, sensitivity analyses, IPCW, multiple imputation).
- 4.3 Distinguish nonresponse bias (failure to participate at the outset) from attrition bias, identify common drivers of nonresponse, and assess how they distort prevalence and exposure-outcome estimates in survey-based research and how far weighting can correct them.
4.1 Attrition Bias in Longitudinal Studies
Longitudinal studies follow participants over time, but not everyone stays. When loss to follow-up is related to both the exposure and the outcome, the resulting estimates become biased. This is attrition bias, a form of selection bias that occurs after study enrollment, gradually reshaping the study sample in ways that distort associations.
The case study below uses the Framingham Heart Study to illustrate what happens when participants lost to follow-up have systematically higher risk profiles than those who remain.
The Framingham Heart Study, initiated in 1948, is one of the most influential longitudinal studies in cardiovascular epidemiology. A study followed over decades in this way faces a particular risk: if participants lost to follow-up have systematically higher risk profiles, for example if they are more likely to smoke, have higher blood pressure, and have lower socioeconomic status, then, because these same factors predict cardiovascular events, the loss of high-risk participants leads to underestimation of risk factor–outcome associations.
The box below states the condition under which attrition produces bias: dropout that depends on both the exposure and the outcome, or on factors strongly related to both.
When Attrition Bias Matters Most
Attrition bias is most problematic when the probability of dropping out depends on both the exposure and the outcome (or factors strongly related to both). If loss to follow-up is random with respect to the exposure-outcome relationship, estimates remain unbiased even with substantial attrition. The critical question is not “how many participants were lost?” but “is loss to follow-up differential with respect to the exposure and outcome?”
4.2 Detecting and Addressing Attrition Bias
Once the condition for bias is clear, the practical task is to judge whether a study's losses meet it and, if they do, how much they matter. The three panels that follow describe the mechanisms that produce differential attrition, the strategies used to quantify and address attrition bias, and the lessons of the Framingham attrition analyses.
Several mechanisms can produce differential attrition in epidemiological studies:
- Illness-related dropout: Participants who become sicker may be unable or unwilling to continue (e.g., advanced cancer patients missing follow-up visits)
- Exposure-related migration: People who experience adverse effects of an exposure may relocate (e.g., workers who develop respiratory symptoms leaving a polluted area)
- Competing mortality: Participants who die from causes related to the study exposure are lost from the sample, removing the most affected individuals
- Socioeconomic barriers: Disadvantaged participants often face transportation, childcare, or work-schedule barriers to continued study participation
Epidemiologists use several strategies to detect and address attrition bias:
- Compare baseline characteristics: Compare those retained versus those lost to follow-up on key exposure, outcome, and confounder variables
- Sensitivity analyses: Conduct worst-case and best-case scenarios for missing outcomes among those lost to follow-up
- Inverse probability of censoring weighting (IPCW): Weight remaining participants to represent those who were lost, based on predictors of attrition
- Multiple imputation: Use statistical models to fill in plausible values for missing outcome data
None of these methods perfectly eliminates attrition bias, but they provide important evidence about its potential magnitude and direction.
Table 8.2 shows illustrative differences of this kind in smoking, systolic blood pressure, body mass index and education between retained participants and those lost to follow-up.
Table 8.2. Illustrative baseline characteristics of retained Framingham participants and of those lost to follow-up.
| Characteristic | Retained Participants | Lost to Follow-Up |
|---|---|---|
| Current smoking (%) | 28% | 42% |
| Mean systolic BP (mmHg) | 132 | 141 |
| Mean BMI (kg/m²) | 26.4 | 28.1 |
| High school education (%) | 72% | 54% |
Values are illustrative.
When those lost to follow-up are more likely to have the exposures and more likely to develop cardiovascular outcomes, the estimated associations between risk factors and heart disease are attenuated in the retained sample.
This part has shown that attrition biases an estimate when dropout depends on both exposure and outcome, as in the Framingham illustration, and that several methods can gauge the magnitude and direction of that bias. The same diagnostic question applies to nonresponse in cross-sectional surveys, the subject of the next part.
4.3 Nonresponse Bias in Cross-Sectional Surveys
Attrition is the longitudinal-cohort version of selection during follow-up. Cross-sectional surveys have the same problem at the moment of recruitment: nonresponse bias occurs at the point of data collection in cross-sectional studies and surveys (Galea & Tracy, 2007). When people who choose not to respond differ systematically from those who do respond, the resulting data do not represent the target population.
The case study below describes evidence from U.S. national health surveys, which are designed to be nationally representative, that survey respondents are healthier than the population they are meant to represent.
The National Health and Nutrition Examination Survey (NHANES) and the National Health Interview Survey (NHIS) are designed to be nationally representative of the U.S. population, but declining response rates in national surveys are a recognized challenge. Keyes et al. (2018) linked respondents from 20 waves of the NHIS (1990 to 2009) to death records and found that, in every year, respondents had lower mortality than the general population, whether or not survey weights were applied (rate ratio 0.86 for men and 0.89 for women): survey respondents are a systematically healthier group, reflecting both nonresponse and the exclusion of people outside private households from the sampling frame. Standard survey weights partially correct for nonresponse, but cannot fully account for unmeasured differences between respondents and nonrespondents.
The three tabs that follow set out the direction nonresponse bias can take, the weighting adjustments that surveys use to correct for it, and the limitations of those corrections.
The direction of nonresponse bias depends on how nonresponse relates to the variables being studied:
- Health behavior surveys: If people with unhealthier behaviors (smoking, sedentary lifestyles) are less likely to participate, prevalence estimates will underestimate the true burden of these behaviors
- Stigmatized conditions: People with HIV, substance use disorders, or mental illness may avoid surveys, leading to underestimation of prevalence
- Health-conscious responders: Conversely, people interested in health may be more likely to participate, inflating apparent health literacy or preventive behavior prevalence
Surveys like NHANES use post-stratification weights to adjust for nonresponse. These weights are calibrated to known population totals (from census data) on variables like age, sex, race/ethnicity, and geography.
The logic: if young men are underrepresented in the survey relative to the census, each young man who did respond receives a higher weight. However, this only corrects for nonresponse that is explained by the weighting variables. If nonresponse is driven by unmeasured factors (like health status itself), weighting alone is insufficient.
Key limitation: Weighting can only correct for nonresponse that is “missing at random” (MAR), where nonresponse depends on observed variables used in the weighting model. If nonresponse is “missing not at random” (MNAR), that is, related to the outcome itself, no amount of weighting will fully remove the bias.
For example, if people who are severely depressed are less likely to complete a mental health survey because of their depression, no adjustment for age, sex, or socioeconomic status can recover the true depression prevalence. This is a fundamental limitation of survey-based research.
This section has shown how attrition during follow-up and nonresponse at data collection reshape a sample after sound recruitment, and why corrections such as weighting cannot fully remove the bias when nonresponse depends on unmeasured factors or on the outcome itself. The reflection and knowledge check that follow close the section, and the next section turns to selection that has already occurred before a study begins, through death, recovery or other losses, and to whether study findings transport to other populations.
Reflection
Consider a longitudinal cohort study of cannabis use and psychotic symptoms among young adults. Over 5 years, 30% of participants are lost to follow-up, and those lost are more likely to be heavy cannabis users. How might this attrition bias the study’s findings? What strategies would you recommend to assess or mitigate this bias?
Key Takeaways
- 4.1 Attrition bias is selection that occurs after enrollment, as loss to follow-up gradually reshapes the cohort. What matters is not how many participants were lost but whether the loss is differential with respect to both exposure and outcome; random loss leaves estimates unbiased even when it is substantial. If those lost from a cohort such as the Framingham Heart Study have higher-risk profiles, risk factor associations are attenuated in the people who remain.
- 4.2 Comparing baseline characteristics of retained and lost participants, worst-case and best-case sensitivity analyses, inverse probability of censoring weighting and multiple imputation all help gauge or reduce attrition bias. None removes it perfectly, but together they indicate its likely magnitude and direction.
- 4.3 Nonresponse bias arises at the point of data collection, when people who decline to take part differ systematically from those who respond, whereas attrition arises during follow-up. Unhealthy behaviors, stigmatized conditions and health-conscious responding shape its direction. Post-stratification weights correct only for nonresponse explained by the weighting variables (missing at random); when nonresponse depends on the outcome itself (missing not at random), no amount of weighting fully removes the bias.
1. Suppose that in a long cardiovascular cohort such as the Framingham Heart Study, participants lost to follow-up had higher rates of smoking and hypertension. This would most likely result in:
2. NHANES uses post-stratification weighting to address nonresponse. This approach is limited because:
3. Which of the following best distinguishes attrition bias from nonresponse bias?
Survivorship Bias & Transportability
Introduction and Overview
Earlier sections covered selection biases that arise during the study: how participants were enrolled, who dropped out. This section addresses biases that arise from a more upstream filter: by the time we study a population, some people are already gone (dead, recovered, lost to history). When the people we end up with are the survival-filtered residue of the people we wish we could study, conventional analysis answers a different question than we think it answers. The section closes by zooming out one step further to ask the related external-validity question: even when our internal estimates are unbiased, do they apply to anyone outside our sample?
Learning Objectives
- 5.1 Explain prevalence–incidence (Neyman) bias and use the P ≈ I × D relationship to predict its direction.
- 6.1 Identify survivorship bias across mortality, treatment persistence, and successful-aging contexts and recognize when conclusions overgeneralize.
- 7.1 Distinguish internal validity from external validity and articulate the conditions under which study findings transport to a new target population.
- 8.1 Choose study designs (incident-case, nested case-control, restricted or reweighted target population) that reduce survival-filter and transportability problems, and use them to judge whether a study has addressed selection-related threats.
5.1 Prevalence-Incidence (Neyman) Bias
One of the most subtle forms of selection bias occurs when we use cross-sectional data to study risk factors for diseases. The problem: cross-sectional studies capture prevalent cases, people who currently have the disease, rather than incident cases, people who are newly developing it. If some cases die quickly while others survive for years, the cross-sectional sample will overrepresent long-surviving cases and underrepresent rapidly fatal ones.
The case study below illustrates the problem with cross-sectional studies of myocardial infarction survivors, in which patients with the most severe risk profiles can die from their initial MI before they are included in a prevalence sample.
Consider a cross-sectional study of myocardial infarction (MI) survivors that examines which risk factors are associated with having had an MI. If patients with the most severe risk profiles, particularly those with extremely high cholesterol or severe hypertension, are more likely to die from their initial MI before they can be included in a prevalence sample, the study will underestimate the strength of these risk factors, because the most affected individuals are already dead and absent from the sample.
The mechanism box below sets out how the bias arises and its direction of the bias for risk factors that increase case fatality and for those that improve survival.
The Neyman Bias Mechanism
Named after Jerzy Neyman (1955), prevalence-incidence bias occurs because prevalent cases are a subset of all incident cases, specifically those who survived long enough to be sampled. If the risk factor under study is associated with case fatality (more severe disease or faster death), it will appear less strongly associated with the disease in cross-sectional data than it truly is. The bias is toward the null (making a real association look weaker or absent) for risk factors that increase case fatality, and away from the null (making it look stronger) for factors that improve survival.
The two cards that follow summarize the relationship between prevalence, incidence and duration, and the incident case designs used to study risk factors.
Neyman bias is the cross-sectional version of the survival-filter problem. The same logic shows up in cohort studies whenever the sample is built around “people still here,” whether that means people still alive, still on treatment, or still showing up for follow-up.
6.1 Survivorship Bias in Cohort Studies
Survivorship bias arises when an analysis is restricted to those who remained after a selective process, so that the survivors stand in for an original population that also included those who were lost. Interactive 8.2 introduces the idea through the WWII bomber-armor problem associated with Abraham Wald, and the case study after it turns to an epidemiological case, studies of long-term non-progressors in the early years of the HIV epidemic.
Walk through the WWII bomber-armor problem (Abraham Wald) that named survivorship bias. Next ▶ advances scenes.
A 7-scene retelling of Abraham Wald's most famous insight: returning bombers mapped with bullet holes, the obvious-but-wrong conclusion, the ghost planes that didn't return, and the rule that armor belongs where the holes AREN'T.
In the early years of the HIV epidemic, before effective antiretroviral therapy, researchers studied “long-term non-progressors”, individuals who remained healthy for years despite HIV infection. Studies of this group identified genetic factors (such as CCR5-delta32 mutations) and immune characteristics associated with slow progression. However, drawing general conclusions about HIV pathogenesis from these survivors was problematic: they represented a highly selected subset of all HIV-infected individuals. The majority of infected individuals had already died, and the survivors differed systematically in ways beyond the factors being studied.
Survivorship bias occurs when we analyze only those who “survived” a process, whether survival means remaining alive, staying in a study, or maintaining a condition, and draw conclusions that we mistakenly generalize to the full original population.
The two panels that follow extend survivorship bias beyond mortality to other forms of selective retention and return to the HIV example with a comparison of long-term survivors and all HIV-infected individuals.
While the term “survivorship” evokes mortality, this bias extends to any selective retention process:
- Treatment persistence studies: Patients who remain on a medication for 2 years are systematically different from those who discontinued; they had fewer side effects, better response, and likely better adherence behaviors. Analyzing only those who persisted overestimates treatment efficacy.
- Cancer survivor cohorts: Studies of quality of life among “cancer survivors” may overestimate well-being because those who died (often with the worst quality of life) are excluded.
- Successful aging studies: Research on cognitive function in elderly cohorts necessarily excludes those who died before reaching old age, potentially missing the very exposures that most strongly impair cognition.
Table 8.3 compares long-term survivors with the estimated population of all HIV-infected individuals on genetic, immunological, demographic and access-to-care characteristics.
Table 8.3. Illustrative characteristics of HIV long-term survivors compared with estimates for all HIV-infected individuals.
| Characteristic | Long-Term Survivors | All HIV-Infected (Estimated) |
|---|---|---|
| CCR5-delta32 heterozygote (%) | 15–20% | ~10% |
| Strong CD8+ T-cell response (%) | 80% | ~40% |
| Younger age at infection (%) | 65% | ~45% |
| Access to care (%) | 90% | ~50% |
Values are illustrative of patterns described in early HIV natural history studies.
Long-term survivors differed on multiple dimensions (genetic, immunological, demographic, and socioeconomic), so findings from survivor cohorts could not be straightforwardly generalized to all people living with HIV.
Survivorship and Neyman-style selection can also be read structurally as collider bias: conditioning on “made it into the sample” opens a non-causal path between exposure and outcome (Cole et al., 2010; Munafò et al., 2018). In causal-diagram terms, being in the sample is a shared effect of exposure and outcome, and selecting on a shared effect links the two even when neither one causes the other. Neyman bias and survivorship bias are about whether the sample we have represents the population we are trying to study. The last topic of the section steps further out: even when the sample does represent its source population, the study's findings may not apply elsewhere. This is the external-validity question that the rest of evidence-based practice depends on.
7.1 Transportability of Study Results
Even a perfectly internally valid study may yield misleading conclusions when its findings are applied to a different population. This is the problem of transportability, also called external validity or generalizability, and it is increasingly recognized as a critical challenge in evidence-based practice (Pearl & Bareinboim, 2014; Westreich, Edwards, Lesko, Stuart, & Cole, 2017).
This is also where the comparison set aside in Section 1 (1.1 What Is Selection Bias?) belongs. Whether the participants resemble the target population is a question of external validity. A sample that does not represent the target population gives descriptive estimates, such as a prevalence, that do not describe the target population, but it produces selection bias in an exposure-outcome association only when entry or retention depends jointly on exposure and outcome. Otherwise its association can be internally valid for the source population and still fail to transport, if effect modifiers are distributed differently in the target population.
Because this argument turns on effect modifiers, the recall box below restates the definition of effect modification from Lesson 6 and works a retrieval question on how the distribution of an effect modifier changes an average effect.
Recall: effect modification
Lesson 6, Section 3 (Inferential Errors and Sources of Ecologic Bias), defined effect modification as variation in the exposure-outcome association across levels of a third variable, on a stated scale: additive (for example, a risk difference) or multiplicative (for example, a risk ratio). An association can be modified on one scale and not on the other, and a standard causal diagram does not show effect modification.
Retrieval question: A drug lowers five-year stroke risk by 3 percentage points in people with diabetes and by 1 percentage point in people without diabetes. The trial population was 70% diabetic, and the target population is 20% diabetic. How does the average effect in the target population compare with the average effect in the trial?
Answer: It is smaller. Weighting the subgroup effects by each population’s mix gives an average reduction of 0.7 × 3 + 0.3 × 1 = 2.4 percentage points in the trial and 0.2 × 3 + 0.8 × 1 = 1.4 percentage points in the target population, provided no other effect modifier differs. A different distribution of an effect modifier changes the average effect even though the effect within each subgroup is the same in both populations.
The case study below illustrates a target population mismatch with the ALLHAT trial, whose participants were adults aged 55 or older with hypertension and at least one other risk factor for coronary heart disease, and whose results clinicians may wish to apply to younger patients with fewer risk factors.
The ALLHAT trial (Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial) found that thiazide-type diuretics were as effective as newer, more expensive antihypertensives in reducing cardiovascular events. However, the trial enrolled 33,357 adults aged 55 or older with hypertension and at least one other risk factor for coronary heart disease (ALLHAT Officers and Coordinators, 2002). Whether the comparison between drug classes holds for younger patients with fewer risk factors is a question of transportability: if age or comorbidity modifies the relative effects of the drugs, the trial result may not apply directly to them.
The box below sets the two questions side by side: internal validity concerns the source population that produced the participants, and transportability concerns a different target population.
Transportability vs. Internal Validity
Internal validity asks: “Apart from random error, does the observed association equal the association in the source population that produced the participants?” Transportability asks: “Would the same causal effect hold in a different target population?” A study can have perfect internal validity yet poor transportability if the study population differs from the target population in ways that modify the effect of interest. Effect modification is the key concept linking internal validity to external generalizability.
The three tabs that follow set out the conditions under which study results are transportable, the role of effect modifiers, and the formal methods for assessing and improving transportability.
Study results are transportable to a target population when:
- The causal mechanism operates the same way in both populations
- There are no effect modifiers whose distribution differs between the study and target populations
- The versions of treatment (or exposure) are comparable across settings
- The outcome measurement is equivalent in both populations
If any of these conditions are violated, the effect observed in the study may not replicate in the target population, not because the study was wrong, but because the populations differ in ways that matter.
Effect modification is the core mechanism that threatens transportability. If the treatment effect differs across subgroups (e.g., by age, comorbidity, or genetics), and the study and target populations have different compositions of these subgroups, the average treatment effect will differ between populations.
Table 8.4 illustrates this with three populations that differ in the proportion of people with comorbidities and in the mean blood pressure reduction under treatment.
Table 8.4. Illustrative mean treatment effects (blood pressure reduction) in three populations with different proportions of people with comorbidities.
| Population | % With Comorbidities | Mean Treatment Effect (BP Reduction) |
|---|---|---|
| ALLHAT trial participants | 78% | −12 mmHg |
| Young adults (<40 years) | 15% | −8 mmHg |
| Elderly with renal disease | 92% | −14 mmHg |
Values are illustrative of how treatment effects can vary across populations due to effect modification.
Researchers have developed formal methods for assessing and improving transportability:
- Inverse probability of selection weighting: Reweight the study sample so that its covariate distribution matches the target population (Westreich et al., 2017)
- Standardization: Estimate the treatment effect within subgroups defined by effect modifiers, then average across the target population’s subgroup distribution
- Sensitivity analysis for transportability: Assess how much unmeasured effect modification would be needed to qualitatively change conclusions
- Target trial emulation: Use observational data from the target population to emulate the trial design and compare results
8.1 Designing Against Survival Filters and Transportability Failures
The design responses to these problems follow from the mechanisms. Neyman bias is avoided by incident case designs: prospective cohorts that follow participants from exposure to new disease onset, case-control studies nested within cohorts that identify cases at diagnosis, or, at minimum, a case definition restricted to newly diagnosed cases within a defined time window. Survivorship bias is reduced by defining the cohort at a point before the selective process acts, such as at diagnosis or at treatment initiation rather than among those still alive or still on treatment. Transportability is addressed by stating the target population in advance and, where its covariate distribution is known, by reweighting or standardizing the study results to it, as described in the Formal Methods tab above. Appraising a study for selection-related threats means asking the same three questions in turn: who got into the sample, who left it, and whether the population the results are applied to differs from the study population in its effect modifiers.
This section has shown how survival filters shape prevalent-case and survivor samples through Neyman bias and survivorship bias, and how even an internally valid result may fail to transport when effect modifiers are distributed differently in the target population. The reflection and knowledge check that follow close the section.
Reflection
A randomized controlled trial of a new diabetes medication was conducted exclusively among patients aged 50–75 at academic medical centers. A community health clinic serving a predominantly young, uninsured population wants to adopt this medication. What factors would you consider when evaluating whether the trial results are transportable to this new setting?
Key Takeaways
- 5.1 Cross-sectional studies capture prevalent cases, who are the subset of incident cases that survived long enough to be sampled, so rapidly fatal cases are underrepresented. Because P ≈ I × D, a risk factor that raises incidence but shortens survival can look weak or absent in prevalence data; the bias is toward the null for factors that increase case fatality and away from the null for factors that improve survival.
- 6.1 Survivorship bias arises when only those who remained after a selective process are analysed and the findings are generalized to the original population. It extends beyond mortality to treatment persistence, cancer survivor cohorts and successful-aging studies, and long-term HIV non-progressors illustrate how survivors differ on many dimensions at once. Structurally, being in the sample is a shared effect of exposure and outcome, and conditioning on it opens a non-causal path.
- 7.1 Internal validity asks whether the observed association equals the association in the source population; transportability asks whether the same causal effect would hold in a different target population. A study can be internally valid yet transport poorly when effect modifiers are distributed differently in the target population, a question that arises when ALLHAT results, obtained in adults aged 55 or older with hypertension and at least one other coronary risk factor, are applied to younger and healthier patients. Results transport when the mechanism, the effect-modifier distribution, the versions of treatment and the outcome measurement are comparable.
- 8.1 Incident case designs, including prospective cohorts and nested case-control studies, avoid Neyman bias, and defining the cohort before the selective process acts limits survivorship bias. Inverse probability of selection weighting, standardization, sensitivity analysis and target trial emulation help assess and improve transportability, while a larger sample improves precision but not transportability.
1. In a cross-sectional study of MI survivors, the association between extremely high cholesterol and MI is weaker than in prospective cohort data. This is best explained by:
2. A study examines quality of life among people who have been on antidepressant therapy for at least 2 years. This study is most susceptible to:
3. The ALLHAT trial results for antihypertensive therapy may not be directly transportable to a younger population because:
4. Which of the following is NOT a valid strategy for improving the transportability of clinical trial results?
Final Assessment
Bringing It All Together
This lesson built a working inventory of selection-related biases in the order they typically arise. An earlier section handled biases that operate at the moment of enrollment, such as Berkson’s bias in hospital-based case-control designs and the healthy worker effect in occupational cohorts. An earlier section turned to losses that happen after enrollment: differential attrition in longitudinal studies and nonresponse in surveys, both invisible unless you ask whether the loss depends on both exposure and outcome. An earlier section tackled the upstream survival filter, prevalence–incidence (Neyman) bias and survivorship bias, then stepped outside the sample altogether to ask the external-validity question: when does an internally valid finding transport?
Read together, these mechanisms form a structured checklist for "the sample is wrong." Combined with an earlier lesson’s vocabulary for "the variables are wrong" and the information-bias inventory ahead in a later lesson, you are building the appraisal toolkit you will use for the rest of the course. The diagnostic question that runs through all three sections is the same: who is in the data, who is missing, and does that missingness depend on the very relationship we are trying to estimate?
The final reflection asks you to apply the full inventory to a single hypothetical study; the 15-question assessment then checks the conceptual material directly. From here, a later lesson turns to information bias, what happens when the people in the study are right but the measurements on them are wrong.
Key Takeaways from this lesson
- Selection bias: when entry into a study, or retention in it, depends jointly on exposure and outcome, the analysis conditions on a common effect of both; this is a structural problem, distinct from confounding, that larger sample sizes cannot fix.
- Berkson’s bias: hospital-based sampling can create an association between conditions that are unrelated in the population, because being hospitalized is itself a selection filter.
- Healthy worker effect: workforce participation pre-selects healthier people, so SMRs below 1.0 against the general population can mask real occupational hazard; internal comparisons, cause-specific and lagged analyses, and population-based controls are the main design remedies for enrollment-stage bias.
- Attrition and nonresponse: what matters is not the absolute rate of loss but whether loss is differential with respect to exposure and outcome; only then does it bias the estimate.
- Neyman bias: cross-sectional samples are filtered by prior survival, biasing risk-factor estimates in directions predictable from P ≈ I × D.
- Survivorship bias: "survivor"-based samples, whether survival means staying alive, staying on treatment or staying in a study, cannot be straightforwardly generalized to the original population.
- Transportability: internal validity does not guarantee external validity; findings travel only when effect-modifier distributions are comparable between study and target populations.
- Diagnostic discipline: for every study, ask who got into the sample, who left, and whether that depends on both exposure and outcome, and whether the target population differs from the study population in its effect modifiers.
Reflection
You are designing a study to estimate the effect of air pollution exposure on childhood asthma in a large metropolitan area. Describe at least three different selection biases that could threaten your study and explain one specific design decision you would make to address each. Be sure to distinguish between biases that affect internal validity versus external validity (transportability).
Final Knowledge Assessment
1. A hospital-based case-control study finds that gallstones are associated with appendicitis. When replicated in a population-based sample, no association is found. This discrepancy is most likely due to:
2. A researcher compares mortality rates of nuclear power plant workers to the general population and finds an SMR of 0.75 for all-cause mortality. The researcher concludes that radiation exposure is not harmful. What is the primary flaw in this reasoning?
3. In a 10-year cohort study of alcohol consumption and liver disease, participants who develop early-stage liver disease are more likely to drop out. This attrition will most likely:
4. A survey on depression prevalence has a 45% response rate. Compared to census benchmarks, respondents are more educated and more likely to be employed. The survey’s depression prevalence estimate is most likely:
5. A cross-sectional study finds no association between a genetic variant and pancreatic cancer. A subsequent cohort study finds a strong positive association. The most likely explanation is:
6. A study of “successful agers” (people over 85 in good health) finds that moderate alcohol consumption is associated with better cognitive function. This finding should be interpreted cautiously because:
7. A clinical trial conducted exclusively in academic medical centers finds that a new chemotherapy regimen reduces mortality by 30%. When the regimen is used in community oncology practices, the benefit is only 15%. This discrepancy most likely reflects:
8. Which of the following correctly describes the relationship between Neyman bias and the formula P = I x D?
9. Inverse probability of censoring weighting (IPCW) is used to address:
10. A researcher wants to study risk factors for sudden cardiac death. Why would a case-control study using prevalent coronary heart disease cases be inappropriate?
11. In a study of occupational asbestos exposure, the SMR for mesothelioma is 8.2 while the SMR for all-cause mortality is 0.85. This pattern indicates that:
12. A survey of health behaviors among university students achieves an 80% response rate. The researcher states this eliminates nonresponse bias. This claim is:
13. HIV long-term non-progressors were found to have distinctive genetic and immune profiles. Generalizing these findings to all HIV-infected individuals is problematic primarily because of:
14. A trial of a new vaccine was conducted in a well-nourished population in a high-income country. When the vaccine is deployed in a malnourished population in a low-income setting, efficacy is substantially lower. The best explanation for this difference is:
15. A researcher argues that increasing the sample size of a clinical trial will automatically make the results more generalizable to other populations. This reasoning is: