Overview

HSCI 230 taught the vocabulary of validity. Its Lessons 7 to 11 defined selection bias, information bias and confounding, named their common forms, and showed how each appears on a causal diagram. In a course on designing research, the vocabulary leads to two practical questions: which decisions in a study protocol prevent each bias, and how the protocol can show, with numbers, how large the bias that remains could be. A protocol often gathers the answers in a subsection titled "threats to validity and mitigation", and this lesson supplies the tools that subsection uses. This section begins with a recap of internal and external validity and the three threats, pairs each threat with the protocol decisions that address it, and checks the distinction between a confounder, a mediator and a collider. It then treats the first threat a protocol plans for, selection bias: when it biases an odds ratio and in which direction, how its size is computed and bounded with sampling fractions, and which decisions about the sampling frame, response, retention and records keep it small.

1.1 Internal Validity, External Validity and the Three Threats

1.1.1 The Three Threats and Where HSCI 230 Teaches Them

A study is internally valid when the association it reports, apart from random error, equals the association in the source population, the population that actually produced the participants. By this we mean that, if the study could be repeated many times in the same source population, its estimates would scatter around the true association rather than around some other value. A study is externally valid when that association also holds in a target population beyond the source. For example, a survey of students at five British Columbia institutions could be internally valid for those students and still tell us little about students in another country, where the causes of loneliness and depression may differ. The two kinds of validity therefore answer different questions: the first asks whether the estimate is right for the people who could have been studied, and the second asks whether it can be carried to other people. Bias is a systematic difference between the estimate and the truth. Because it is systematic, it pushes every repetition of the study in the same direction, and so, unlike random error, it does not shrink as the sample grows: a larger sample gives a more precise estimate of the wrong value. Three threats account for most bias in observational research, and HSCI 230 treated each of them in depth. Selection bias arises when entry into the study, or remaining in it, depends on both the exposure and the outcome. Information bias arises when the exposure, the outcome or the covariates are measured with error. Confounding arises when the groups compared differ in other causes of the outcome. All three make the estimate wrong for the people actually studied, so all three are threats to internal validity, whereas a concern that a correct estimate may not transfer to another population is a question of external validity. Table 7.1 serves as the whole recap. Where a definition is unfamiliar, the HSCI 230 lesson named in its third column is the place to review it, and the last column names the tools this lesson adds for each threat.

Table 7.1. The threats to internal validity, how each arises, where HSCI 230 teaches it, and what this lesson adds.

1.1.2 Bias Is Decided When the Protocol Is Written

Most of the decisions that determine whether a study is biased are made before any data are collected. The sampling frame and the recruitment method decide who can enter the study. The instruments, their mode of administration and the timing of measurement decide how much misclassification there will be. The list of variables to be measured decides which confounders can ever be controlled, because an analysis can adjust only for what was recorded. Restriction and matching are design choices, and stratification and adjustment, although carried out in the analysis, must be planned in the protocol so that the necessary variables are collected and the analysis is not chosen after the results are seen. Temporal threats belong in the plan as well. Reverse causation arises when the exposure and the outcome are measured at the same time, or the outcome begins first, so that the outcome may cause the exposure rather than the reverse. A protocol prevents it by measuring the exposure before the outcome, for example in a cohort that excludes people who already have the outcome at baseline, or by asking when the outcome began. It assesses it with a secondary analysis that excludes people whose outcome began before the exposure, and when the order cannot be established it describes the estimate as a cross-sectional association and states reverse causation as a limitation. Table 7.2 pairs each threat with the protocol decisions that prevent it and the analysis steps that check or correct it.

Table 7.2. Protocol decisions and analysis steps for each threat to validity.

The protocol therefore carries a plan for each threat, often written in a subsection titled threats to validity and mitigation. Each section of this lesson ends with the decisions it contributes to that subsection, and Section 4 shows a complete example. Before the first of those threats is taken up, one distinction from HSCI 230 needs to be secure, because selection bias and confounding both depend on it.

1.2 Telling a Confounder from a Mediator and a Collider

A third variable can be associated with both the exposure and the outcome in three different ways, and the data alone cannot tell them apart. In each case the variable will be correlated with the exposure and with the outcome, so a table of correlations looks the same for all three. What separates them is the direction of the arrows in a causal diagram, which records what the investigator believes causes what, and this is why the diagram has to be drawn from subject-matter knowledge rather than read off the data.

A confounder is a common cause of the exposure and the outcome. By this we mean that it sits upstream of both, with one arrow running from it to the exposure and another running from it to the outcome. Because it causes both, it creates an association between them even when the exposure has no effect, which is what a backdoor path means. A confounder is therefore controlled in the protocol, by design or in the analysis, so that this non-causal association is removed. A mediator is different, because it lies on the path from the exposure to the outcome: the exposure changes the mediator, and the mediator in turn changes the outcome. It carries part of the effect the study wants to estimate, so adjusting for it would remove part of that effect. For this reason it is left out of the adjustment set when the target is the total effect. A collider is a common effect of the exposure and the outcome (or of their causes), so that the arrows point into it rather than out of it. Left alone, a collider does no harm. An analysis that conditions on it, however, for example by analyzing only the people who completed a survey, creates or distorts an association, because among the people selected in this way the exposure and the outcome become linked through their shared influence on being selected. Timing matters as well. A covariate measured after the exposure can be a mediator, while the same characteristic measured before the exposure could be a confounder, which is why a protocol states when each covariate is measured.

Most people who have studied these threats can define them, and the most common error is to describe a variable on the causal pathway as a confounder. The ungraded check below targets that confusion. It takes two or three minutes, and three of its four items use the Campus Connection Study and a shift-work cohort that recur in this lesson.

The four items turn on the direction of the arrows. Financial strain is a confounder, perceived social support and body mass index measured after shift work are mediators, and completing the survey is a collider on which an analysis of completers conditions. That last structure is the one behind selection bias, which the rest of this section develops.

2.1 When Selection Biases an Odds Ratio

HSCI 230 described selection bias as conditioning on a common effect. When both the exposure and the outcome influence who takes part, restricting the analysis to participants creates an association between them, or distorts one that exists. Figure 7.1 shows this collider structure in general and in the study that serves as this lesson's running example, the Campus Connection Study, a worked example that this lesson shares with Lesson 8. That study is a survey of loneliness and depressive symptoms among students aged 18 to 29 at five public post-secondary institutions in British Columbia.

The diagram shows that selection bias is possible, and the key word in its structure is "both". HSCI 341 Lesson 2 Section 3 called selection that depends on the outcome the formal criterion for selection bias, and that criterion holds for a descriptive estimate such as a prevalence, which is biased whenever cases and non-cases take part at different rates. A measure of association such as the odds ratio behaves differently, because it is a comparison between groups rather than a single proportion. Selection that depends on the exposure alone, or on case status alone, leaves the odds ratio unbiased and costs only precision. For example, a case-control study deliberately samples cases at a far higher rate than non-cases, yet its odds ratio is valid, because the over-sampling of cases affects the exposed and the unexposed cases equally and cancels when the two groups are compared. In the same way, if exposed people respond less often than unexposed people but the shortfall is the same among cases and non-cases, the comparison within each exposure group is untouched. Selection that depends on the outcome therefore biases the odds ratio only when it depends on the outcome differently in the exposed and the unexposed. By this we mean that participation depends on the exposure and the outcome jointly, so that one particular combination, such as exposed cases, is missing more often than the separate effects of exposure and outcome would predict. The sampling fractions introduced next turn this condition into a number.

The direction of the bias can often be reasoned out without any numbers by asking which cell of the two-by-two table is under-represented. Suppose that in a case-control study nearly all cases take part, but exposed controls are less willing to take part than unexposed controls. The controls who take part then contain too few exposed people, so the exposure odds among controls are too low, and the observed odds ratio, which divides the cases' exposure odds by the controls', is too large: for a harmful exposure the bias is away from the null. If instead exposed controls were more willing to take part, the controls would look too exposed and the odds ratio would be biased toward the null. If exposed and unexposed controls took part at the same rate, the odds ratio would be unbiased, however low that rate. The same reasoning applies to loss to follow-up in a cohort: when exposed people who develop the disease are the ones most often lost, the risk or rate among the exposed is underestimated, and the ratio is biased toward the null, while a loss that depends on exposure only leaves the ratio unbiased and costs only precision.

One consequence is a frequent exam trap, worth stating on its own.

To judge how large such a bias could be, the reasoning about under-represented cells has to be made quantitative, which is the purpose of the sampling fractions in the next subsection.

2.2 Measuring Selection Bias with Sampling Fractions

2.2.1 The Odds Ratio of the Sampling Fractions

Sampling fractions show how large a selection bias is and in which direction it runs. Imagine the source population arranged in the usual two-by-two table, with exposure in the columns and disease in the rows. Upper-case letters denote counts in the source population and lower-case letters denote the counts that end up in the study. Table 7.3 sets out the source population in this form.

Table 7.3. The source population as a two-by-two table of exposure and disease, with counts in upper-case letters.

Each cell has a sampling fraction, the proportion of that cell of the source population that entered the study and stayed in it to the analysis. Equation 7.1 defines the four fractions, each as the study count divided by the source count for one cell.

Equation 7.2 combines the four fractions into a single quantity, the odds ratio of the sampling fractions (OR_sf). The observed odds ratio equals the source-population odds ratio multiplied by the odds ratio of the sampling fractions:

Three consequences follow, and together they give the direction of the bias. When OR_sf equals 1, the observed odds ratio is unbiased, even when the four fractions differ from each other. When OR_sf is above 1, the observed odds ratio is larger than the true one, which for a harmful exposure is a bias away from the null value of 1. When OR_sf is below 1, the observed odds ratio is smaller, which for a harmful exposure is a bias toward the null. The same information can be expressed as sampling odds. The selection odds among the exposed (sf₁₁ ÷ sf₂₁) compare how readily exposed cases enter the study with how readily exposed non-cases enter it, and the selection odds among the unexposed (sf₁₂ ÷ sf₂₂) make the same comparison for the unexposed. Each of these odds therefore describes how strongly selection favours or disfavours cases within one exposure group. There is no selection bias in the odds ratio when these two odds are equal. In other words, cases may be over-sampled or under-sampled, but if they are over-sampled or under-sampled to the same degree among the exposed and the unexposed, the distortion cancels. This is the joint-dependence condition of 2.1 written as a formula. The result applies exactly to the odds ratio. For a risk ratio it holds approximately when the outcome is rare, because the risk ratio and the odds ratio are then close to each other.

Worked Example 7.1 applies Equation 7.2 to non-response in a source population of 10,000 people, first when non-response depends on exposure alone and then when it also depends on disease, so that both halves of the condition can be seen in numbers.

The first scenario confirms that unequal response by exposure alone gives OR_sf = 1, and the second shows a value of 0.80 pulling the odds ratio toward the null, exactly as the under-represented cell (exposed cases) predicts. In this example the true fractions were known. In a real study they are not, which is the problem the next part addresses.

2.2.2 Bounding the Bias When the Fractions Are Unknown

The true sampling fractions are almost never known, because the source population is not observed cell by cell. The formula is still useful as a sensitivity analysis, by which we mean an analysis that asks how the result would change under assumptions the data cannot check. The investigator assigns plausible fractions to the four cells, computes OR_sf, and divides the observed odds ratio by it to see what the estimate would have been without the selection. Plausible values can come from the records of who responded, from response rates reported by similar studies, or from the investigator's knowledge of who is hard to recruit. Assigning one set of values, or a few sets that span the plausible range, is a deterministic analysis; drawing the fractions from distributions and repeating the calculation many times is a probabilistic one, which Section 4 describes as part of quantitative bias analysis. Worked Example 7.2 applies a deterministic analysis to a case-control study of daycare attendance and childhood respiratory disease.

Because the corrected value is the observed odds ratio divided by OR_sf, a value of OR_sf above 1 lowers the estimate and a value below 1 raises it, which is a quick check that the correction moves in the direction the reasoning about missing cells predicts. Calculator 7.1 applies Equation 7.2 to any four sampling fractions and an observed odds ratio. Its presets include the daycare study of Worked Example 7.2 and the two Campus Connection cases of Worked Example 7.3 in 2.3.

A bounding calculation shows how large the bias could be, but it does not remove it. Selection bias can be corrected fully in only two situations. In the first, the factors that drive selection are measured, precede both exposure and outcome, and have a known distribution in the source population, so that they can be handled like confounders, for example by weighting. By this we mean that each participant is given more weight when people like them were less likely to take part, so that the weighted sample resembles the source population again. For example, if older students responded half as often as younger students, and age is the only thing driving response, each older respondent can be counted twice. In the second situation, the investigator can identify and measure a variable that carries the whole dependence of selection on exposure and outcome (sometimes called a bias breaker) and can obtain unbiased estimates of its distribution in the source population. Such a variable breaks the bias because, once it is accounted for, participation no longer depends on the exposure and the outcome together. Both situations require information collected by design, which is why the protocol decisions in the next subsection include what to record as well as how to recruit.

2.3 Protocol Decisions That Prevent Selection Bias

The aim of the design is to keep OR_sf close to 1: selection may depend on exposure or on outcome, but it should not depend on their combination. The decisions fall into four groups. The sampling frame decides who can enter the study, and a frame that is independent of the exposure and the outcome, such as a registrar's enrolment list, allows probability sampling with a known denominator. Response decisions, such as a neutral invitation and repeat contact with non-responders, reduce the room for response to depend on the exposure and the outcome. Retention decisions do the same for loss to follow-up in a cohort. Records of who was invited, who responded and who was lost let the analysis compare responders with non-responders, weight the data and choose the sampling fractions for a bounding analysis. Recruiting extra participants or replacing those who drop out does not help, because it changes how many people are analyzed, not who is missing.

The four tabs below give the detail behind each group of decisions. Open each tab in turn to see what a protocol states about the sampling frame, response, retention and what to record. The Response tab also recalls the strategies for raising response from HSCI 341 Lesson 3 and ends with a retrieval question to try before you reveal the answer.

Each decision in the tabs either keeps participation from depending jointly on the exposure and the outcome or records the information needed to check whether it did.

Worked Example 7.3 applies the four groups of decisions to the Campus Connection Study and then uses Equation 7.2 to bound the selection bias that its expected response rate of 15% leaves room for.

The two Campus Connection cases show the point of 2.1 in numbers: unequal response leaves the odds ratio unbiased as long as the reductions in response combine multiplicatively, and only a further drop among students who are both lonely and depressed produces bias.

Section 1 has shown that selection biases an odds ratio only when participation depends jointly on the exposure and the outcome, that the odds ratio of the sampling fractions measures the size and direction of that bias, and that a protocol keeps it small through its sampling frame, recruitment message, contact schedule, retention procedures and records of the people who do not take part. The key takeaways below summarize these points. The knowledge check that follows closes the section, and its last two questions apply the sampling fractions to a study that recruits through social media.

Overview

HSCI 341 Lesson 5 described a measurement in terms of its sensitivity (Se), the probability that a person who truly has a characteristic is classified as having it, and its specificity (Sp), the probability that a person who truly lacks it is classified as lacking it. HSCI 230 Lesson 9 showed the consequence for a study: misclassification that is the same in the groups being compared (non-differential) tends to pull a measure of association toward the null, and misclassification that differs between them (differential) can push it either way. This section makes both statements quantitative and ties each calculation to a protocol decision. It computes the table a study would observe under non-differential exposure misclassification, shows why a protocol favours a specific outcome definition, and explains why differential misclassification has no predictable direction and how a protocol prevents it. It then runs the calculation in reverse to correct an observed table, plans the validation sub-study that supplies the sensitivity and specificity a correction needs, and inflates the sample size so that misclassification does not leave the study underpowered.

3.1 Non-Differential Exposure Misclassification

The first calculation shows what non-differential misclassification of a binary exposure does to the two-by-two table a study observes. Misclassification of exposure is non-differential when the sensitivity and specificity of the exposure measurement are the same among cases and non-cases, which is the condition Equation 7.3 states:

Each observed cell is then a mixture of correctly classified people and people who belong in the other exposure column. Among the cases, the observed exposed count contains the truly exposed cases who were correctly classified and the truly unexposed cases who were wrongly classified as exposed. The same mixing happens among the non-cases. Table 7.4 gives each observed count as a mixture of two true counts, weighted by Se_E and Sp_E.

Table 7.4. Observed counts under non-differential exposure misclassification, in terms of the true counts, Se_E and Sp_E.

Because the mixing blends the exposed and unexposed columns in the same proportions for cases and non-cases, the two columns become more alike and the odds ratio moves toward 1, provided Se_E + Sp_E is greater than 1. By this we mean that some truly unexposed people are counted among the exposed, and some truly exposed people are counted among the unexposed, so each column ends up containing a little of the other. If the exposure raises the risk of disease, the observed exposed group is diluted with lower-risk unexposed people and the observed unexposed group is contaminated with higher-risk exposed people. The difference between the two columns therefore shrinks, much as mixing a little of two different paint colours into each other makes both look more alike. The condition on Se_E + Sp_E matters because a measurement with a sum of exactly 1 is no better than a coin toss, and one with a sum below 1 does worse than chance and would reverse the association rather than weaken it. Misclassifying exposure moves people between the exposure columns and leaves the case and non-case totals unchanged, which is a useful check on any calculation and the property that makes back-correction possible in 4.1. Error rates that look modest, of 10% to 20%, can attenuate an association considerably. Worked Example 7.4 applies the formulas of Table 7.4 to a study whose true table is known, to show the size of that attenuation.

The example confirms both properties of non-differential exposure error: the odds ratio moves toward 1, and the row totals are preserved. Calculator 7.2 reproduces the example, with a second preset for better measurement, so that other values of Se_E and Sp_E can be tried.

The same mixing logic applies when it is the outcome, rather than the exposure, that is measured with error, but the two kinds of outcome error do very different damage.

3.2 Misclassification of the Outcome

Outcome misclassification has an asymmetry that matters for design. When the outcome is measured with error, each observed risk is p′ = Se × p + (1 − Sp) × (1 − p), the true cases the measure detects plus the people without the outcome whom it wrongly flags. Suppose the true risk of disease is 0.10 among the exposed and 0.05 among the unexposed, a risk ratio of 2.0. If the outcome measure misses some cases (sensitivity 0.80) but never labels a healthy person as a case (specificity 1), both observed risks are multiplied by 0.80, giving 0.08 and 0.04, and the risk ratio stays at 2.0. If instead the measure finds every case (sensitivity 1) but wrongly labels 5% of healthy people as cases (specificity 0.95), the observed risks become 0.145 and 0.0975, and the risk ratio falls to 1.49. The two kinds of error behave differently because of what they do to each risk. Missed cases remove the same fraction of cases from each group, so both risks shrink in proportion and their ratio is preserved. False positives, by contrast, add roughly the same number of spurious cases to each group, whatever the group's true risk. Adding the same amount to two numbers makes them closer in relative terms, so the ratio is diluted toward 1. The dilution is greatest when the outcome is rare, because a fixed number of false positives is then large compared with the small number of true cases. A protocol therefore favours a highly specific outcome definition, for example a confirmed diagnosis or a validated case-finding algorithm, even at some cost in sensitivity. That cost is real: a specific definition misses some true cases, or enrols them only later once they are confirmed, so the study has fewer cases and less precision and may need a larger sample, although with perfect specificity the ratio itself stays unbiased.

In a case-control study the same logic applies to the case definition: false-positive cases bias the odds ratio toward the null, so diagnoses are confirmed before enrolment. The reason is that a false-positive case is really a non-case, so its exposure history looks like that of a control, and including it among the cases makes the cases and controls more alike. Sensitivity and specificity estimated in the general population cannot be used directly to correct a case-control study. This is because a case-control study enrols cases and controls at very different rates, for example most of the people diagnosed with a disease but only a small fraction of the people without it. Sampling at different rates changes the proportions of true and false positives among those enrolled, so the error rates among the people actually in the study are no longer the ones measured in the general population.

Subsections 3.1 and 3.2 assumed that the error rates were the same in the groups being compared. When they are not, the predictable pull toward the null disappears.

3.3 Differential Misclassification

3.3.1 Why the Direction Cannot Be Assumed

When the accuracy of the exposure measurement differs between cases and non-cases, or the accuracy of the outcome measurement differs between exposed and unexposed people, the misclassification is differential. Equation 7.4 states the condition for differential exposure misclassification:

Its direction cannot be assumed, and its size cannot be predicted without knowing the error rates in each group. The pull toward the null in 3.1 depended on the two rows being blended in the same proportions. Once one group is measured more accurately than the other, that symmetry is lost, and the result depends on which group is made to look more exposed. Recall bias, in which cases search their memories for exposures more thoroughly than controls, is the familiar example. A parent whose child has a birth defect, for instance, may think back over every medication taken in pregnancy, while the parent of a healthy child has no reason to. HSCI 230 Lesson 9 described observer and detection bias as other examples. Because the error rates differ between groups, differential error cannot be corrected by a single pair of sensitivity and specificity values; a correction would need a separate pair for each group.

Worked Example 7.5 shows why the direction cannot be assumed. It applies the formulas of Table 7.4 one row at a time, with each group's own error rates, to the true table of Worked Example 7.4. Watch which group ends up looking more exposed than it really is.

The same true table therefore gives an odds ratio above or below the truth depending on which group reports more accurately, so the direction of differential error has to be worked out, not assumed. To work out the direction in a given study, the group-specific values are needed, and they come from a validation sub-study that samples cases and non-cases separately (step 3 of the plan in 4.2). In each group, the sensitivity is the number who report exposure among those whom the reference standard shows to be exposed, and the specificity is the number who report no exposure among those whom it shows to be unexposed, exactly as in HSCI 341 Lesson 5. When the two groups' values differ, the misclassification is differential, and a group that has both the higher sensitivity and the lower specificity will appear more exposed than it truly is. Figure 7.2 contrasts the two kinds of misclassification.

3.3.2 Protocol Decisions That Prevent Differential Error

Because differential error has no fixed direction and cannot be corrected with a single pair of values, the protocol concentrates on preventing it. Table 7.5 lists the decisions that do so and what each prevents.

Table 7.5. Protocol decisions that prevent differential misclassification.

The second decision in Table 7.5 holds the mode of administration constant. The mode of administration is the way questions reach a respondent: on the web, on paper, by telephone or in person. HSCI 341 Lesson 3, Section 1 (Planning a Questionnaire), compared these methods by response rate, cost and the potential for interviewer bias. A mode effect is a difference in survey results that arises from the mode by which the data were collected, and it has two components. A selection effect concerns who answers by each mode: people who respond online can differ from people who respond by telephone in age, health and the outcomes under study, so a comparison across modes can mix different groups of people. A measurement effect concerns how the mode changes the answers of the same person: self-completed modes, on paper or online, tend to produce greater disclosure of sensitive behaviour, such as substance use or symptoms of depression, than interviewer-administered modes, in which the wish to give a socially acceptable answer is stronger. The measurement effect is the reason a protocol uses one mode for every participant. When a study must offer more than one mode, it records the mode of each response so that results can be compared by mode in the analysis. HSCI 207 Lesson 8, Section 2.1 (Mode Effects), treats the topic in more detail and is optional reading.

Non-differential misclassification of a binary exposure moves the odds ratio toward the null by an amount that Table 7.4 allows the analyst to compute, while differential misclassification has to be prevented by design. The calculation in Table 7.4 can also be run in reverse, which is the subject of 4.1.

4.1 Back-Correcting an Observed Table

When the sensitivity and specificity of the exposure measurement are known, or can be assumed, the true table can be recovered from the observed one. Because misclassifying exposure leaves the non-case total m₀ unchanged, the true count of exposed non-cases is given by Equation 7.5:

The same reasoning applied to the case total m₁ gives Equation 7.6 for the exposed cases.

The unexposed counts follow by subtraction from the totals. As an illustration, apply the formulas to the observed table of Worked Example 7.4. With the true values Se_E = 0.80 and Sp_E = 0.90, they return the original 90, 70, 210 and 630, and the odds ratio of 3.86. With Se_E = 0.90 and Sp_E = 0.95 applied to the same observed table, they return an odds ratio of 3.03. The corrected estimate therefore depends heavily on the assumed error rates, and for some combinations the formulas give a zero or negative cell, which shows that the assumed values are incompatible with the data. For these reasons the correction is reported over a range of plausible values, and those values come from a validation study in a population like the study population. Calculator 7.3 contains both versions of the example.

A back-corrected table is only as accurate as the sensitivity and specificity used to compute it. The next subsection describes how a protocol plans the validation sub-study that supplies those values.

4.2 Planning a Validation Sub-Study

A validation sub-study measures a subset of participants twice: once with the study's instrument and once with a reference standard that is accepted as correct or close to it. It supplies the sensitivity and specificity (or, for a continuous measure, the relationship between measured and true values) that a correction needs. Two directions of estimation should be kept apart. A validation estimate runs from truth to measurement. By this we mean that it starts with people whose true status is known and asks how the instrument classified them, for example the probability of being classified as exposed among those who are truly exposed, which is the sensitivity. A correction runs the other way, from measurement to truth. It starts with what the instrument recorded and asks what is really there, for example the probability of being truly exposed among those classified as exposed, which is a predictive value. The distinction matters because a predictive value depends on prevalence as well as on the error rates. For example, a measure with a specificity of 0.90 used in a population where almost nobody is exposed will produce many false positives relative to true positives, so most of the people it classifies as exposed are not, even though its sensitivity and specificity are unchanged. Equation 7.7 sets the two directions side by side.

4.2.1 Five Planning Steps

A protocol plans the sub-study in five steps: it chooses a reference standard, decides between internal and external validation, decides who is sampled, decides how many to verify, and states how the results will be used. The fourth step is the one most often done wrongly, because sensitivity is estimated only among people who truly have the condition, so the number to verify depends on how many of them the sub-study will contain. Expand each of the five panels below to see what the protocol states at that step, including the sample size formula for step 4.

Taken together, the five panels describe a sub-study whose results can be trusted and used: an independent reference standard, a population to which the results transfer, a sample that allows group-specific estimates, a size set by the true positives it contains, and a stated analysis.

Worked Example 7.6 applies the five steps to the PHQ-2 in the Campus Connection Study, including the sample size calculation of the fourth step.

The example shows the slip that step 4 guards against: the 34.4 participants with depression are the number of true positives needed, and the number of interviews is found only after dividing by the planning prevalence.

4.2.2 Regression Calibration for a Continuous Exposure

The fifth step names regression calibration as the method for a continuous variable. For a continuous exposure measured with error, such as dietary intake from a food-frequency questionnaire, regression calibration is the analogue of back-correction (Rosner, Willett & Spiegelman, 1989). The problem it solves is the continuous version of the one in 3.1: random error in a measured exposure spreads the measured values out more widely than the true values, so the slope of the outcome on the measured exposure is flatter than the true slope, a pattern called attenuation. Regression calibration works in two stages. First, in the validation sub-study, the reference measurement is regressed on the study measurement (and on any other covariates in the main model). This regression describes what a person's true exposure is likely to be, given what the questionnaire recorded. For example, people who report very high intakes on a food-frequency questionnaire tend, on average, to have true intakes somewhat closer to the middle, and the fitted equation captures that pull. Second, the fitted equation is used to predict a calibrated exposure value for every participant in the main study, and the outcome model is fitted with the calibrated values in place of the measured ones. Because the calibrated values no longer carry the extra spread, the slope moves away from the attenuated value toward the true association. Its standard error must be adjusted for the uncertainty in the calibration step, for example by bootstrapping. The method assumes that the measurement error is non-differential and that the calibration equation transfers from the sub-study to the whole sample. A protocol that plans regression calibration therefore plans an internal validation sub-study with both measurements, and HSCI 410 provides the regression skills to carry it out.

Validation and correction address the bias that misclassification causes. Misclassification also reduces power, and the last subsection turns to the sample size a protocol needs when its outcome measure is imperfect.

4.3 Inflating the Sample Size for Misclassification

Sample size formulas assume that the proportions entered into them are the proportions the study will observe. With an imperfect outcome measure, the observed proportions are given by Equation 7.8:

With non-differential error the difference p₁′ − p₂′ is smaller than p₁ − p₂, so a study sized with the true proportions would be underpowered. The recall box below summarizes how HSCI 341 Lesson 2 computed the sample size for comparing two proportions; try its retrieval question before you reveal the answer.

Power depends mainly on how far apart the two proportions are rather than on how common the outcome is (the proportions also affect the variance, but less), so observed proportions that are higher than the true ones still call for a larger sample. Misclassification enters through the last of those inputs, the two expected proportions, and the protocol therefore computes the observed proportions first and enters them into the two-proportion formula. Worked Example 7.7 applies Equation 7.8 to a cohort comparison and to the Campus Connection Study.

Both cases show the required sample growing because the observed proportions move closer together. Calculator 7.4 reproduces both cases, with a third preset for a perfect test.

Two further inflation factors appear in protocols. The first is the design effect from unequal weighting. When respondents carry weights that vary, because strata were sampled at different rates or because the weights were adjusted for non-response as in the Campus Connection Study, the effective sample size is smaller than the number of respondents. By this we mean that a weighted sample of 1,000 respondents carries the statistical information of a smaller unweighted sample, because a few heavily weighted people have a large influence on the estimate and make it less stable. Kish's approximation gives this design effect as 1 + CV², where CV is the coefficient of variation of the weights, that is, their standard deviation divided by their mean. The more the weights vary, the larger the design effect, and the required sample size is multiplied by it. For example, weights with a coefficient of variation of 0.5 give a design effect of 1.25, so a study needs a quarter more respondents. The second is the variance inflation factor, 1 ÷ (1 − R²), where R² is the proportion of the variance of the exposure explained by the other covariates in a regression model. The idea is that, once the analysis adjusts for those covariates, only the part of the exposure that they do not explain is left to estimate its effect, so the more of the exposure they explain, the less information remains. It is the factor in the multicollinearity formula of HSCI 230 Lesson 11 Section 2, and it applies only when a regression-adjusted analysis, taught in HSCI 410, is planned: adjustment for covariates that are correlated with the exposure inflates the variance of the exposure coefficient by this factor, so the sample size is multiplied by it.

Section 2 has shown that misclassification affects a study in two ways. It biases the estimate, which the protocol limits by measuring in the same way in every group and assesses with back-correction and a validation sub-study, and it reduces power, which the protocol offsets by sizing the study with the observed proportions. The key takeaways below summarize these calculations. The knowledge check that follows closes the section, and its last two questions apply outcome misclassification and the planning of a validation sub-study to a screen for generalized anxiety.

Overview

HSCI 230 Lessons 7 and 11 defined confounding, showed it on causal diagrams, and illustrated Simpson's paradox, in which an association reverses when a third variable is taken into account. In a protocol, the control of confounding is a sequence of decisions, and this section follows that sequence. It sets out what makes a variable a confounder and why the causal diagram, not a checklist of criteria, settles the question. It compares the four ways to control a confounder and works through restriction and matching, uses the causal diagram to choose the adjustment set, and carries out a stratified analysis by hand with the Mantel-Haenszel estimator. It ends by separating confounding from two phenomena that can be mistaken for it in a stratified table: effect modification and the non-collapsibility of the odds ratio.

1.3 What Makes a Variable a Confounder

A confounder is a common cause of the exposure and the outcome, or a proxy for such a cause, that is not on the causal pathway from the exposure to the outcome. This is the definition the diagnostic check in Section 1 used. The three classical conditions follow from it. A confounder is associated with the outcome independently of the exposure, because it causes the outcome or is a proxy for a cause. It is associated with the exposure in the source population, because it causes the exposure or shares a cause with it. And it is not caused by the exposure, so it does not lie on the path from exposure to outcome, and it is not a consequence of the outcome.

The three conditions are necessary but not sufficient. By this we mean that every true confounder meets them, but a variable can meet them and still not be a confounder, so they can rule a variable out but cannot, on their own, rule it in. A mediator satisfies the first two conditions and fails the third, which is why it is so often mistaken for a confounder: it is associated with the outcome and with the exposure, and only knowledge of the order of events shows that the exposure causes it. A subtler case is a variable measured before the exposure that is caused by two unmeasured factors, one that also causes the exposure and one that also causes the outcome. Such a variable is associated with the exposure through the first factor and with the outcome through the second, and the exposure does not cause it, so it meets all three conditions. Yet it is a collider, because two arrows point into it. Left alone, it blocks the path between the two unmeasured factors, but adjusting for it opens that path and so creates a non-causal association between the exposure and the outcome. Because the diagram of this structure looks like the letter M, the resulting bias is called M-bias. In each of these cases the causal diagram, which shows the direction of each arrow, settles the question that the three conditions leave open.

Two further distinctions affect what a protocol states. A population confounder is a variable known from previous research to confound the association in the target population. Because its role is established before the study begins, the protocol plans to control it whatever the study data show, even if in this particular sample it happens to change the estimate very little. A sample confounder, by contrast, appears to confound in the study data only. This can happen by chance, when a variable that is unrelated to the exposure in the population happens to be unevenly spread between the exposed and the unexposed in the people who were studied. The protocol states in advance how such variables will be judged. The reason is that, if every variable that happens to change the estimate were added to the adjustment set, the set would be shaped by the data rather than by prior knowledge, and the investigator could, knowingly or not, choose the adjustments that give the most appealing result.

Worked Example 7.8 introduces a study of carriage of Streptococcus pneumoniae and childhood respiratory disease in which a measured variable, RSV infection, meets these conditions. Its crude and stratified tables are used throughout this section and in Section 4.

In the STREP example, RSV is associated with both the exposure and the outcome, and the crude odds ratio overstates the association found within each RSV stratum. Once a protocol has identified such a confounder, it has to decide how to control it, which is the subject of 5.1.

5.1 Four Ways to Control a Confounder

A protocol chooses, for each confounder, one of four approaches or a combination of them. The first two, restriction and matching, act on who is studied. The last two, stratification and regression adjustment, act on how the data are analyzed and require that the confounder be measured well. Table 7.7 compares the four approaches by how each works, what it costs and what the protocol states about it. Stratification is carried out by hand in 6.1, and regression adjustment is taught in HSCI 410, so this subsection develops the two design-stage approaches.

Table 7.7. Four approaches to controlling a confounder.

5.1.1 Restriction

Restriction prevents confounding by admitting only one level of the confounder. Restricting a study of cervical cancer to women removes confounding by gender completely. Restricting a study to people aged 60 to 69 reduces confounding by age without removing it, because age still varies within the band. When the confounder is binary, admitting the low-risk level is usually preferred, because the exposure effect is then easier to detect and effect modification by the restricted variable is avoided. The other side of that coin is a second cost: because only one level is studied, neither the effect of the restricted variable nor effect modification by it can be examined. If the association is in fact much stronger at the excluded level, for example in children rather than adults, restriction discards the most important finding, and the variable was a modifier to be described rather than a confounder to be removed. In the STREP example, restricting to children without RSV gives an odds ratio of 3.21. That estimate is free of confounding by RSV and describes only children without RSV, which is the cost of restriction: the protocol trades the breadth of its conclusions for their internal validity.

5.1.2 Matching

Matching, the second design-stage approach, makes the distribution of a confounder the same in the groups being compared. HSCI 230 Lesson 4, Section 4 (Comparability, Analysis and Reporting), introduced matching as a way to make controls comparable to cases, and noted that a matched design requires a matched analysis. Its glossary defined overmatching as matching on a variable on the causal pathway, an effect of the exposure, or a variable unrelated to confounding, which reduces efficiency or biases the estimate toward the null. The consequences of matching differ by design, because in the two designs a different group is chosen to resemble another. In a cohort study, unexposed people are chosen to resemble the exposed on the matched variable, before the outcome occurs. The exposed and unexposed groups then have the same distribution of the confounder, so the confounder can no longer differ between the groups being compared, and matching removes confounding by it. In a case-control study, controls are chosen to resemble the cases on the matched variable, after the outcome has occurred. The groups being compared here are cases and controls, and because the matched variable is associated with the exposure, making the controls resemble the cases on it also makes the controls' exposure resemble the cases'. For example, if cases are older and older people are more often exposed, age-matched controls will be older and therefore more often exposed than a random sample of controls would be. This is a selection bias that only a matched or stratified analysis removes. A matched cohort can therefore report the crude comparison, with a stratified or matched analysis adding only precision, whereas a matched case-control study must include the matched variable in its analysis. Matching takes two forms. Frequency matching makes the overall distribution of the matched variable the same in cases and controls, for example the same share in each age group, and is analyzed by stratifying on that variable; because every level stays in the study, effect modification by the matched variable can still be examined, which restriction prevents. Pair matching links each case to one or more specific controls; with one control per case, only the pairs that differ in exposure carry information, and they give the matched odds ratio and McNemar's test in Equation 7.9.

The three tabs below set out these consequences design by design. Open each tab to compare matching in cohort studies, in case-control studies and in matched pairs; the Matched pairs tab gives Equation 7.9 and a numerical example of the matched odds ratio and McNemar's test.

The tabs show that matching never removes the need to think about the analysis: a matched case-control study must be analyzed as matched, and the protocol states that analysis alongside the matching. They also explain why overmatching costs precision. If the matching variable is closely tied to the exposure, for example matching on employer when exposure to a solvent depends on the employer, a case and its control usually share the same exposure, so most pairs are concordant and carry no information. The few discordant pairs then give an imprecise estimate, possibly biased toward the null, and no confounding is removed, because the matching variable was not a confounder.

Restriction and matching act on who is studied, and both depend on knowing which variables are confounders, since matching on a variable that is associated with the exposure but is not a confounder is overmatching. The causal diagram supplies that knowledge, and 5.2 shows how it determines the adjustment set.

5.2 Choosing the Adjustment Set from a Causal Diagram

A causal diagram, or directed acyclic graph (DAG), records the investigator's assumptions about what causes what. HSCI 230 Lesson 7 taught how to draw one and how conditioning on a collider or a mediator biases an estimate. For a protocol, the diagram has a specific job: it determines the adjustment set, the variables that must be measured and controlled to estimate the effect of the exposure, using the backdoor criterion (Greenland, Pearl & Robins, 1999). The criterion is applied in four steps.

1. Draw the diagram from subject-matter knowledge before seeing the study data, including unmeasured variables.
2. Set aside every arrow that leaves the exposure, and every variable that the exposure causes.
3. List the remaining paths that connect the exposure to the outcome. Each one that begins with an arrow into the exposure is a backdoor path.
4. Choose a set of measured variables that blocks every backdoor path, without including colliders on those paths (unless the path is also blocked by another variable in the set) and without including descendants of the exposure.

Figure 7.3 presents the backdoor criterion as a story in seven scenes, from the causal front door to the closing of the backdoor by conditioning on the confounder.

Figure 7.4 then applies the four steps to a simplified causal diagram for the Campus Connection Study.

In this simplified diagram there are two backdoor paths from loneliness to depressive symptoms, one through financial strain and one through gender, and the adjustment set {financial strain, gender} blocks both. Perceived social support is caused by loneliness, so it is excluded: adjusting for it would estimate only the part of the effect that does not pass through support. Completing the survey is a collider; it cannot be adjusted away, and Section 1 dealt with it as selection bias. Living arrangement affects loneliness and, in this diagram, affects depressive symptoms only through loneliness, so it opens no backdoor path and is not needed for control of confounding; adjusting for a variable that affects only the exposure adds imprecision without removing bias. If the investigators believed that living arrangement also affected depressive symptoms directly, they would draw that arrow and add the variable to the set. The protocol's real diagram contains more variables than this one. Its value lies in making each of these judgements visible to reviewers before the data are collected, and in ruling out the tempting rule of adjusting for everything associated with both the exposure and the outcome.

Once the diagram has named the confounders, the analysis has to control them. For one or two categorical confounders this is usually done by stratification, which 6.1 carries out by hand.

6.1 Stratified Analysis with the Mantel-Haenszel Estimator

6.1.1 Pooling the Strata

The Mantel-Haenszel procedure stratifies the data by the levels of the confounder, computes the odds ratio within each stratum, and pools the strata into one adjusted estimate with a test of whether that estimate differs from 1 (Mantel & Haenszel, 1959). In practice it is accompanied by a test of whether the stratum-specific values are similar. It is the analysis that a protocol usually plans when it has one or two categorical confounders. Equation 7.10 gives the odds ratio within one stratum, Equation 7.11 pools the strata into the Mantel-Haenszel estimate, Equation 7.12 tests whether the stratum-specific values are similar, and Equation 7.13 tests whether the pooled estimate differs from 1.

The pooled estimate weights each stratum by its size, so it is not the simple average of the stratum odds ratios, and every product is divided by its own stratum total. Worked Example 7.9 applies Equation 7.11 to the two RSV strata of Worked Example 7.8.

Calculator 7.5 reproduces the calculation, reports the homogeneity test of Equation 7.12, and has a second preset in which the strata differ.

6.1.2 Judging Confounding: Change in Estimate and Direction

The Mantel-Haenszel estimate supplies the adjusted value that the change-in-estimate approach needs. This approach compares the crude odds ratio (OR_c) with the odds ratio adjusted for a candidate variable (OR_a). By the common convention, a change of more than 10 percent is taken as evidence that the variable confounds the association, and a protocol that uses another threshold names it in advance as a choice. Three rules keep the check honest. The crude value is the baseline. For ratio measures the change is computed on the log scale, (ln OR_c − ln OR_a) ÷ ln OR_c, because a halving and a doubling are changes of the same size on that scale. By this we mean that an odds ratio of 2.0 and an odds ratio of 0.5 describe associations of equal strength in opposite directions, yet on the ordinary scale the first is 1.0 above the null and the second only 0.5 below it. Taking logarithms makes the two distances equal (ln 2.0 = 0.693 and ln 0.5 = −0.693), so a change measured on this scale means the same thing whether the association is harmful or protective. For a logistic model this is the percentage change in the exposure coefficient, because the coefficient is the log odds ratio, and it is the form that HSCI 410 Lesson 3 uses. And the check is applied only to variables that the diagram allows to be confounders, because adjusting for a mediator or a collider also changes the estimate. In the STREP example the Mantel-Haenszel estimate in Worked Example 7.9 is 3.36, and the change from the crude value is (ln 5.44 − ln 3.36) ÷ ln 5.44 = 28% on the log scale, so RSV meets the criterion and the diagram supports it as a confounder. A protocol names the diagram as the primary basis for the adjustment set, states the threshold it will use for variables whose role is uncertain, and fixes both before the analysis.

The comparison of the crude and adjusted values also gives the direction of the confounding. In the STREP example the crude odds ratio (5.44) is further from 1 than the adjusted one (3.36), so the confounding was positive: RSV is more common among carriers and raises the risk of CRD, and the crude estimate overstated the association. Confounding can also be negative. When a confounder is more common among the exposed but lowers the risk of the outcome, or is less common among the exposed but raises it, the crude estimate is closer to 1 than the adjusted one and hides part of the association, as the negative-confounding preset of Interactive 7.1 shows. In both cases the adjusted estimate is the one to report, and the change-in-estimate check is computed in the same way. With the crude value as the baseline, negative confounding gives a negative change. For a crude odds ratio of 2.0 and an adjusted value of 2.5, (ln 2.0 − ln 2.5) ÷ ln 2.0 = (0.693 − 0.916) ÷ 0.693 = −0.32. The sign shows the direction, and it is the size of the change, 32%, that is compared with the 10% threshold, so this variable confounds the association. A change of −3%, by contrast, is well within 10%, and the variable would not be judged a confounder.

Interactive 7.1 simulates a study of 2,000 people in which a third variable, C, raises the risk of the outcome. Its sliders change the strength of confounding and of effect modification, so that the crude, stratum-specific and Mantel-Haenszel odds ratios can be compared as each changes. Its effect-modification slider also previews the subject of 6.2.

In the simulator, raising the strength of confounding separates the crude odds ratio from the stratum values, while raising the effect-modification multiplier separates the stratum values from each other. That second pattern is not confounding at all, and 6.2 explains how to tell the two apart.

6.2 Effect Modification and Non-Collapsibility

6.2.1 Effect Modification

Effect modification is present when the association between exposure and outcome differs across levels of a third variable. By this we mean that the exposure genuinely does more, or less, in one group than in another, as when a drug works better in younger patients than in older ones. This makes it a different kind of thing from confounding. Effect modification is a property of the causal system, a real feature of how the exposure acts, and so it is described and reported. Confounding, in contrast, is a distortion produced by comparing groups that differ in other causes of the outcome, and so it is a bias to be removed. A single variable can be a confounder, an effect modifier, both or neither, and the analysis has to ask each question separately. Whether effect modification is present also depends on the scale of measurement: on the additive scale it is judged with the risk difference, and on the multiplicative scale with a ratio measure such as the risk ratio or odds ratio. The three cards below give the formal definitions. Click each card to see how interaction is defined on the additive scale and on the multiplicative scale, and what the terms synergism and antagonism mean.

The cards show that the multiplicative scale is the one the Mantel-Haenszel homogeneity test examines, while the additive scale bears most directly on public health, because it describes how many cases a combination of exposures adds.

The two scales can give different answers for the same data. Suppose that among younger people the risk of an outcome is 0.12 with an exposure and 0.04 without it, and among older people it is 0.24 with the exposure and 0.08 without it. The risk ratio is 3.0 in both age groups, so there is no effect modification on the multiplicative scale. The risk difference is 0.08 among younger people and 0.16 among older people, so on the additive scale the exposure adds twice as many cases per 100 people in the older group, because their baseline risk is twice as high. A program that can reach only a fixed number of exposed people would prevent more cases among the older group, which is the additive-scale question. A protocol therefore states in advance the scale on which it will assess effect modification, and the measure it will use to answer a public health question about where to act.

On the multiplicative scale, effect modification appears in a stratified analysis as stratum-specific odds ratios that differ, and the protocol has a fixed rule for that case.

6.2.2 Non-Collapsibility of the Odds Ratio

The odds ratio has a property that can mimic confounding. Even when a third variable is unrelated to the exposure, so that it cannot confound, the crude odds ratio can differ from the stratum-specific odds ratios whenever that variable affects the outcome and the outcome is common (Greenland, Robins & Pearl, 1999). This property is called non-collapsibility. By this we mean that the odds ratio for the combined table is not a weighted average of the odds ratios in the strata, so collapsing the strata together can change its value even though nothing has been distorted. It is a mathematical feature of odds, not a bias, and it arises because odds grow much faster than risks as an outcome becomes common. Suppose 100 exposed and 100 unexposed people are studied in one stratum, with 80 and 50 cases, and 90 exposed and 90 unexposed in a second stratum, with 30 and 10 cases. The exposure is equally common in both strata, so the stratifying variable is unrelated to it. The odds ratio is 4.0 in each stratum and the Mantel-Haenszel estimate is 4.0, but the crude odds ratio from the combined table (110 of 190 exposed and 60 of 190 unexposed are cases) is 2.98. The change-in-estimate rule would flag a confounder where there is none. Two checks separate the explanations. The first asks whether the third variable is associated with the exposure, which it must be to confound (here it is not). The second uses the risk ratio, which is collapsible: without confounding, its crude value is a weighted average of the stratum-specific values. Here the stratum risk ratios are 1.60 and 3.00 and the crude risk ratio is 1.83, which lies between them. When the outcome is rare the odds ratio approximates the risk ratio and non-collapsibility is negligible.

Three patterns in a stratified table therefore call for three different reports. When the stratum odds ratios are similar to each other but differ from the crude value, and the third variable is associated with the exposure, the variable confounds the association and the pooled Mantel-Haenszel estimate is reported. When the stratum values differ from each other more than chance would explain, the variable modifies the effect and each stratum is reported separately. When identical stratum values differ from the crude odds ratio although the third variable is unrelated to the exposure, the difference reflects non-collapsibility rather than confounding.

Section 3 has set out the decisions a protocol makes about confounding. The protocol identifies confounders with a causal diagram, controls each one by restriction, matching, stratification or adjustment, and separates confounding from effect modification and from the non-collapsibility of the odds ratio. The key takeaways below summarize these decisions. The knowledge check that follows closes the section, and its last two questions ask for the adjustment set and a Mantel-Haenszel analysis in a cohort study of first-year students.

Overview

Sections 1 to 3 dealt with biases that a protocol can prevent or correct with information it collects. Some confounders cannot be measured at all, and this section shows how a protocol reasons about them and then brings the whole lesson together. It begins with a short orientation to the model-based methods that later courses teach and the limitation they share. It then gives three computations that show how much an unmeasured confounder could change a result: external adjustment with a bias factor, a planned sensitivity analysis and the E-value. It ends with the threats to validity and mitigation subsection of a protocol, with a complete example for the Campus Connection Study.

7.1 Methods That Scale to Many Confounders

Stratification works well for one or two confounders. The difficulty is that every added confounder multiplies the number of strata, because each stratum must hold one combination of levels of all the confounders at once. With five confounders of three levels each there are 3 × 3 × 3 × 3 × 3 = 243 strata, many of them nearly empty. A stratum that contains no exposed people, or no cases, cannot contribute to a comparison, and the Mantel-Haenszel estimator therefore loses most of its information. Model-based methods summarize the confounders more efficiently. Instead of comparing people only within exact combinations of the confounders, they assume a pattern, such as a model in which each confounder shifts the outcome by a fixed amount, and so they can use every participant at once. Later courses teach them, and a protocol for a study with one or two confounders often plans a stratified analysis and names the model-based method that a fuller analysis would use. Table 7.8 summarizes what each method does, its key assumption and where it is taught, and it completes, at the level of orientation, the confounding-control methods that HSCI 341 Lesson 1 Section 4 previewed: propensity scores, regression and difference-in-differences.

Table 7.8. Model-based methods for many confounders: what each does, its key assumption and where it is taught.

In the STREP example, logistic regression of CRD on STREP and RSV gives an odds ratio of 3.35, and stratification on the propensity score gives 3.36, identical to the Mantel-Haenszel estimate because with a single binary confounder the propensity score takes only two values. The methods differ in how they handle many covariates. Except for instrumental variables, and difference-in-differences for confounders that are stable over time, they share one limitation: they adjust only for what was measured. A propensity score, for example, is the probability of exposure given the measured covariates, so matching, stratifying or weighting on it can balance only what went into it. Several methods that agree with each other, as these three do for STREP, therefore do not rule out an unmeasured confounder, because they all assume there is none. Only a design in which the exposure is assigned rather than chosen removes this limitation. In a randomized trial, chance decides who is exposed, so measured and unmeasured confounders are balanced between the groups on average. When people choose their exposure, as when they decide to join a program, an observational protocol can only bound unmeasured confounding with the methods that follow, and a randomized trial is the design that would settle the question.

A protocol therefore also needs a way to reason about the confounders it could not measure, which the next three subsections take up.

7.2 External Adjustment for an Unmeasured Confounder

Every protocol leaves some confounders unmeasured, because they were unknown, too costly or too sensitive to ask about, or because the data source does not record them. The protocol cannot remove this residual confounding, but it can name the likely candidates, state the direction of the bias each would cause, and plan a calculation that shows how strong a confounder would have to be to change the conclusion. The first such calculation applies when outside sources describe the confounder. When a confounder U was not measured in the study but its prevalence and its association with the disease are known from other sources, the observed estimate can be divided by a bias factor (Schlesselman, 1978):

The bias factor can be read in two parts. The numerator, p₁ × (OR_UD − 1) + 1, describes how much the confounder raises the average risk of disease among the exposed, given how common it is in that group, and the denominator does the same for the unexposed. By this we mean that if U is present in 40% of the exposed and multiplies the odds of disease by about 4.2, the exposed group as a whole carries more of U's extra risk than an unexposed group in which only 13% have it. The bias factor is the ratio of these two amounts, and so it measures how much of the observed difference between the exposure groups is really due to the unequal distribution of U. Dividing by it removes that share. The formula is exact for risk ratios when the confounder is binary and its risk ratio with the disease is the same among the exposed and the unexposed; it is approximate for odds ratios, and the approximation is best when the disease is rare. In a case-control study the two prevalences describe the source population, which the controls represent. Worked Example 7.10 applies Equation 7.14 to the STREP data as if RSV had not been recorded, so that the result can be checked against the stratified answer of Section 3.

The comparison of the adjusted and observed values shows the direction of the bias. In Worked Example 7.10 the bias factor is above 1 and the adjusted value (3.38) is closer to 1 than the observed one (5.44), so the unmeasured confounder had biased the observed estimate away from the null. A bias factor can also be below 1. This happens when the confounder protects against the disease (OR_UD below 1) but is more common among the exposed, or raises the risk but is less common among the exposed. Suppose an observed odds ratio is 2.0, and a protective factor with OR_UD = 0.5 is present in 50% of the exposed and 20% of the unexposed. Then B = [0.50 × (0.5 − 1) + 1] ÷ [0.20 × (0.5 − 1) + 1] = 0.75 ÷ 0.90 = 0.83, and the adjusted odds ratio is 2.0 ÷ 0.83 = 2.4. Dividing by a number below 1 raises the estimate, so here the confounder had masked part of the association and biased the observed estimate toward the null. The same reading applies to a protective exposure: if the adjusted value lies closer to 1 than the observed one, the observed estimate exaggerated the protection. Calculator 7.6 reproduces Worked Example 7.10, with a second preset for a strong confounder that is far more common among the exposed.

External adjustment gives one corrected estimate for one set of outside values, and the agreement in Worked Example 7.10 depends on the accuracy of those values. A protocol therefore also plans for the possibility that they are wrong, which is the subject of 7.3.

7.3 Planning a Sensitivity Analysis

When no single set of external values is trustworthy, the protocol plans a sensitivity analysis that varies the bias parameters over plausible ranges and reports how the estimate changes. The same logic applies to all three threats. For selection bias the parameters are the sampling fractions of Section 1, for misclassification they are the sensitivity and specificity of Section 2, and for an unmeasured confounder they are the prevalences and the strength used in external adjustment. Quantitative bias analysis is the general name for these methods (Lash, Fox & Fink, 2009). It ranges from simple analyses with one set of fixed values, through grids of fixed values, to probabilistic analyses that draw the parameters from probability distributions and repeat the correction many times. The three forms differ in how they treat uncertainty about the bias parameters. A simple analysis asks what the estimate would be if the parameters took one particular value. A grid asks the same question for several combinations of values, so the reader can see which assumptions matter. A probabilistic analysis goes further: it assigns each parameter a range of plausible values with some more likely than others, draws a value at random thousands of times, and reports the spread of the corrected estimates, which shows both the most likely corrected value and how uncertain it is. HSCI 230 Lesson 12 interprets sensitivity analyses that published studies report; a protocol plans one in five steps.

1. Name each threat to be analyzed and the direction in which it would move the estimate.
2. Define its bias parameters.
3. State where plausible values will come from: published studies, national surveys, pilot data or the study's own validation sub-study.
4. Choose the form of the analysis: a grid of fixed values (deterministic) or distributions (probabilistic).
5. State how the results will be reported, including the values at which the conclusion would change.

Worked Example 7.11 applies these steps to an unmeasured confounder of the STREP and CRD association, with a grid of fixed values for the prevalence and the strength that enter the bias factor of Equation 7.14.

Reading the grid means looking for the cells where the adjusted estimate approaches 1 and describing how strong, and how unequally distributed, the confounder would have to be to get there, which is the sentence the fifth step asks the report to contain. A grid of this kind can be summarized further, which leads to the E-value.

7.4 The E-Value

VanderWeele and Ding (2017) summarized this kind of analysis in a single number. The E-value is the minimum strength of association, on the risk-ratio scale, that an unmeasured confounder would need to have with both the exposure and the outcome, beyond the measured confounders, to explain away an observed association completely. By "explain away" we mean that, once the confounder was accounted for, the adjusted risk ratio would fall all the way to 1. The E-value therefore turns the grid of the previous subsection into a single threshold. A large E-value means that only a very strong confounder, strongly tied to both the exposure and the outcome, could account for the result, so the finding is robust to unmeasured confounding. A small E-value, close to 1, means that even a weak confounder could do so. The E-value does not say whether such a confounder exists; that judgement still rests on subject-matter knowledge of what was left unmeasured. Equation 7.15 gives the E-value for a risk ratio above 1.

For a risk ratio of 2.0, E = 2.0 + √(2.0 × 1.0) = 3.41, so a confounder would need a risk-ratio association of at least 3.4 with both the exposure and the outcome to explain the result away. An odds ratio or hazard ratio can be treated as a risk ratio when the outcome is rare; when the outcome is common (above about 15%), the square root of the odds ratio approximates the risk ratio (VanderWeele & Ding, 2017). In a protocol the E-value is a planning tool. The investigator computes it for the effect the study is designed to detect, compares it with the strength that the named unmeasured confounders could plausibly have, and states in advance that it will be reported for the estimate and for the confidence limit closer to 1. Worked Example 7.12 carries out this planning step for the Campus Connection Study.

The example uses both versions of the E-value, for the estimate and for the confidence limit closer to 1, and compares them with the plausible strength of a named confounder. Calculator 7.7 reproduces it, with presets for two other estimates.

To judge whether a named confounder could explain an estimate away, compare both of its associations with the E-value. Its association with the outcome is its risk ratio with the outcome. Its association with the exposure can be gauged roughly from the prevalences used in external adjustment, as the ratio of its prevalence in the two exposure groups. Suppose an estimate has a risk ratio of 1.6, so its E-value is 1.6 + √(1.6 × 0.6) = 2.58, and a named confounder triples the risk of the outcome but is present in 40% of the exposed and 20% of the unexposed, a ratio of 2.0. Its association with the outcome exceeds the E-value, but its association with the exposure falls short, so on its own it could not explain the association away completely. If either association falls below the E-value, the confounder cannot do so.

External adjustment, a planned grid and the E-value give a protocol three ways to show how much an unmeasured confounder could change a result. The last subsection places these results, with those of Sections 1 to 3, in the protocol subsection on threats to validity and mitigation.

7.5 Writing the Threats to Validity and Mitigation Subsection

The subsection brings the four sections of this lesson together for one study. It usually sits in the methods of a protocol, after the analysis plan and sample size. A reviewer reading it should be able to see that each threat has been identified for this particular study, that its likely direction has been reasoned out, and that the protocol contains specific steps to prevent it and to measure what remains. A strong subsection is specific to the study, quantitative where the tools of this lesson allow, consistent with the causal diagram, sampling plan and sample size elsewhere in the protocol, and candid about the threats that cannot be removed. Table 7.10 lists what the subsection states for each threat and the tools from this lesson that supply it.

Table 7.10. The elements of a threats to validity and mitigation subsection and the tools from this lesson that supply them.

The subsection is usually organized by threat (selection, information, confounding, and any design-specific or temporal threat), in about half a page to a page. It is drafted from material already in the protocol: the causal diagram, the sampling plan and sample size, and the instruments chosen. It does not need to repeat the full sample size or analysis plan; it refers to them. Where this lesson gives a calculation (an odds ratio of the sampling fractions, a sample size inflated for misclassification, an E-value for the effect the study is designed to detect), the subsection reports the result.

Missing data cut across the threats: unit non-response and loss to follow-up are selection problems, while item missingness affects information and the measured confounders, so the subsection states the missingness it expects, the mechanism it assumes and the method it will use. The three mechanisms that HSCI 230 Lesson 11 described decide which method is valid, and they are worth committing to memory. Try the retrieval question in the recall box before you reveal the answer.

Worked Example 7.13 gives the complete subsection for the Campus Connection Study. It draws on the selection decisions and bound of Worked Example 7.3, the size of the validation sub-study in Worked Example 7.6 and the E-value of Worked Example 7.12, and it adds a plan for missing data that applies the mechanisms above.

Each entry in the example states the five elements of Table 7.10 for one threat, in sentences specific to this study rather than general definitions. The final knowledge assessment applies the same elements to other studies, including a cohort of first-year students.

Section 4 has shown how a protocol reasons about confounders it cannot measure, and how the threats to validity and mitigation subsection gathers the tools of all four sections into one part of the protocol. The key takeaways below summarize these points. The knowledge check that follows closes the section, and its last two questions apply the bias factor and the E-value to an unmeasured confounder.