# Lesson 9: Information Bias & Data Quality

*Companion-podcast transcript • Sarah & Kiffer*  
*Office Hours episode to listen to after working through the lesson*

---

**Sarah:** Welcome back to Office Hours. I'm Sarah.

**Kiffer:** And I'm Kiffer. This episode goes with Lesson nine, Information Bias and Data Quality.

**Sarah:** It's meant for after you've worked through the lesson. We'll add some perspective on how epidemiologists think about measurement error in practice, some critique of the rules of thumb that get repeated about it, a few of the questions students find thorny, and some extra worked examples, including a couple that are harder than the ones on the lesson page.

**Kiffer:** There are a few places where we ask you to work something out before we give the answer. When that happens, you'll hear a few seconds of quiet. Use them to try the problem yourself, and pause the audio if you'd like more time.

**Sarah:** We have five questions. First, what equal errors in both groups actually do to an odds ratio. Second, how unequal errors can create an association out of nothing, and whether a study can call its estimate conservative. Third, what happens when one group is simply looked at more closely. Fourth, why a single blood pressure reading weakens a real association, and what rounding does near a threshold. And fifth, whether an objective instrument protects a study from information bias.

**Kiffer:** Question one. If the errors are the same in cases and controls, what happens to the odds ratio, and does it matter which kind of error it is? Let's build a made-up case-control study. There are five hundred cases and five hundred controls. Among the cases, two hundred and fifty were truly exposed. Among the controls, one hundred were truly exposed. Sarah, what's the true odds ratio?

**Sarah:** The odds of exposure in the cases are two hundred and fifty to two hundred and fifty, so one. In the controls it's one hundred to four hundred, so 0.25. One divided by 0.25 gives a true odds ratio of four.

**Kiffer:** Good. Now suppose exposure comes from a questionnaire with a sensitivity of eighty percent and a specificity of ninety percent, and it performs the same way in cases and controls. So this is nondifferential misclassification. What odds ratio would the study observe? Take a few seconds.

*(Pause)*

**Sarah:** I'll try it. Eighty percent of the two hundred and fifty exposed cases get recorded as exposed, which is two hundred. Eighty percent of the one hundred exposed controls is eighty. So the recorded table has two hundred exposed and three hundred unexposed cases, and eighty exposed and four hundred and twenty unexposed controls. Two hundred times four hundred and twenty, divided by three hundred times eighty, gives 3.5.

**Kiffer:** That's toward the null, which is the direction you'd expect. Before we accept it, look at the four hundred truly unexposed controls. What does a specificity of ninety percent say about them?

**Sarah:** Oh. Ten percent of them get recorded as exposed, and I left them out. I only used the sensitivity. So in the controls, forty false positives join the eighty true positives, giving one hundred and twenty recorded as exposed and three hundred and eighty unexposed.

**Kiffer:** Good catch. Now apply the same ten percent to the unexposed cases and tell us what happens to them.

**Sarah:** Ten percent of the two hundred and fifty unexposed cases is twenty-five, so the cases have two hundred and twenty-five recorded as exposed and two hundred and seventy-five unexposed. Two hundred and twenty-five times three hundred and eighty, divided by two hundred and seventy-five times one hundred and twenty, is about 2.6. So the observed odds ratio is about 2.6, compared with a true value of four.

**Kiffer:** That's right, and your first attempt shows something worth noticing. Losing twenty points of sensitivity alone took the odds ratio from four to 3.5. If I run it the other way, with perfect sensitivity and ninety percent specificity, the odds ratio falls to about 3.1. A ten point loss of specificity did more damage than a twenty point loss of sensitivity.

**Sarah:** Because most of the controls are unexposed. A small share of a big group is a lot of people, and they all land in the exposed row of the controls, where they make the comparison group look more exposed than it is.

**Kiffer:** Exactly. When an exposure is uncommon, specificity usually matters more than sensitivity. When an outcome is uncommon in a cohort study, the same logic applies to specificity of the outcome measure. It's a useful thing to check when you read a validation study, because authors sometimes report only the sensitivity.

**Sarah:** Question two. How can unequal errors create an association where none exists? I'd like to set up the harder example for this one. Imagine a made-up case-control study of a birth defect, with four hundred mothers of affected infants and four hundred mothers of unaffected infants. In truth, one hundred mothers in each group took a particular over-the-counter medicine during pregnancy, so the true odds ratio is exactly one.

**Kiffer:** And I'm assuming the exposure comes from an interview with each mother after the birth.

**Sarah:** Right. Mothers of affected infants search their memories harder, so they recall the medicine with a sensitivity of ninety percent. Mothers of unaffected infants recall it with a sensitivity of seventy percent. A few mothers in both groups wrongly report taking it, so the specificity is ninety-five percent in each group. What odds ratio does the study report, and is this misclassification differential or nondifferential? Take a few seconds.

*(Pause)*

**Kiffer:** The specificity is the same, which can make it look nondifferential, but the sensitivity differs between cases and controls, and that's enough to make it differential. Now the arithmetic. Among case mothers, ninety percent of one hundred is ninety, plus five percent of the three hundred unexposed, which is fifteen, so one hundred and five recorded as exposed and two hundred and ninety-five unexposed. Among control mothers, seventy plus fifteen gives eighty-five exposed and three hundred and fifteen unexposed.

**Sarah:** So the odds ratio is one hundred and five times three hundred and fifteen, divided by two hundred and ninety-five times eighty-five.

**Kiffer:** Which comes to about 1.32. The medicine has no effect, and the study reports about thirty-two percent higher odds of the birth defect among mothers who took it.

**Sarah:** And that's the size of association that often gets published as a modest finding. Nothing about it looks suspicious on its own.

**Kiffer:** That's why real examples of recall bias are usually caught through a pattern that makes no biological sense. One national component of the INTERPHONE study of mobile phones and glioma, published by Hepworth and colleagues in 2006, is the classic one. Among regular users, the odds ratio was 1.24 for tumours on the side of the head where people said they held their phone, and 0.75 on the opposite side. The authors suggested that this pair of opposite findings might be due to recall bias.

**Sarah:** Because a phone can't protect one side of your head while it harms the other. People who knew where their tumour was seem to have shifted their reported phone use toward that side.

**Kiffer:** That leads to the disagreement I wanted to have with you. Plenty of discussion sections say something like, our exposure measure was imperfect, but the errors were nondifferential, so our estimate is conservative and the true effect is probably larger. I think that argument is reasonable when its conditions hold. For a binary exposure, with errors that don't depend on the outcome or on each other, the expected effect of the error is to pull the estimate toward one.

**Sarah:** I'd push back on that pretty hard. A 2005 paper in the International Journal of Epidemiology made the point directly. The toward-the-null rule describes what happens on average over many hypothetical repetitions of a study. Any single study also has random error, so its estimate can land above the true value by chance even when the misclassification is nondifferential. The authors concluded that nondifferential misclassification doesn't justify a claim that the observed estimate must be an underestimate.

**Kiffer:** That's fair, and it's the strongest form of your argument. I'd still say the direction of the expected bias is useful information. If a study reports an odds ratio of 2.5 with a nondifferential measure, I learn something different than if the error was likely differential.

**Sarah:** I agree it's useful. My worry is that the word conservative gets used as a reassurance, and the conditions rarely get checked. Was the exposure really binary? Were the errors really independent of the outcome? A recall-based measure in a case-control study almost never meets that standard.

**Kiffer:** So where do we land? I'd say the practical rule is that a study can describe the expected direction of bias, but it should state the conditions that direction depends on, and it should back the claim with a bias analysis that plugs in plausible values of sensitivity and specificity, the way we just did by hand.

**Sarah:** I can agree with that. And if the authors can't name plausible values for sensitivity and specificity, they aren't in a position to call the estimate conservative.

**Kiffer:** Question three. What happens when one group is looked at more closely than the other? Here's another made-up cohort. Ten thousand adults with diabetes and ten thousand without are followed for five years, and the outcome is a recorded diagnosis of depression. In truth, six percent of each group develops depression. People with diabetes see their doctors often and are screened for depression routinely, so eighty percent of their cases get diagnosed. In the comparison group, only fifty percent of cases get diagnosed.

**Sarah:** So it's the recorded diagnosis that's being counted, and the true depression is the same in both groups.

**Kiffer:** Right. What risk ratio would the study report? Take a few seconds.

*(Pause)*

**Sarah:** Six percent of ten thousand is six hundred true cases in each group. With diabetes, eighty percent of six hundred is four hundred and eighty recorded cases. Without diabetes, fifty percent of six hundred is three hundred. Four hundred and eighty divided by three hundred is 1.6. So the study would report sixty percent higher risk of depression with diabetes, when the true risk ratio is one.

**Kiffer:** That's surveillance bias, and the arithmetic shows why it's so persistent. Nobody misrecorded anything. Every recorded diagnosis is a real case. The bias comes entirely from the share of true cases that got found in each group.

**Sarah:** And it doesn't have to work alone. If diabetes really did raise the risk of depression by twenty percent, the same detection gap would multiply that by 1.6, and the study would report about 1.9.

**Kiffer:** So if you were designing this study, what would you change to take the detection gap out?

**Sarah:** The cleanest fix is to make detection equal. Give everyone the same structured depression questionnaire at the same times, whether they have diabetes or not. If that finds ninety percent of true cases in both groups, it gives five hundred and forty recorded cases in each, and the risk ratio goes back to one. Another option is to compare people with diabetes with people who have a different condition that brings them to the doctor just as often.

**Kiffer:** That second idea is the same logic as restricting to people with similar screening histories, which is how researchers have tested whether surveillance explains an association. And there's a warning sign you can check in any paper. If an association is strongest for mild or early disease, and weak for severe disease, differential detection is a likely part of the story, because severe disease tends to get found whoever is looking.

**Sarah:** Question four is about a quieter kind of error. Why does a single measurement weaken a real association, even when nobody is misclassified?

**Kiffer:** Because one reading of something like blood pressure or salt intake is a person's usual level plus that day's fluctuation. The fluctuation spreads the measured values out, and the slope gets flattened. The lesson's regression dilution ratio, lambda, is the share of the variation in a single measurement that reflects real differences between people. The observed slope is lambda times the true slope.

**Sarah:** And a real example of how much this matters?

**Kiffer:** MacMahon and colleagues, in The Lancet in 1990, combined nine prospective studies with about four hundred and twenty thousand people. After correcting for regression dilution, a usual diastolic blood pressure ten millimetres of mercury lower went with at least fifty-six percent less stroke. The corrected associations were about sixty percent greater than in earlier uncorrected analyses.

**Sarah:** Let me set up a made-up version with salt. A study uses one twenty-four hour dietary recall per person and finds that each extra gram of sodium a day goes with 0.5 millimetres of mercury higher systolic blood pressure. In a subsample, people completed two recalls on different days, and the correlation between their two recalls was 0.25. That correlation is the estimate of lambda. Kiffer, what's the corrected slope?

**Kiffer:** You divide the observed slope by lambda. 0.5 divided by 0.25 is two millimetres of mercury per gram, four times what the single recall showed.

**Sarah:** Now the harder step. Suppose a new study averages four recalls per person. What lambda should it expect, and what slope would it observe? Take a few seconds.

*(Pause)*

**Kiffer:** The tempting answer is four times 0.25, which is one, meaning no dilution at all. That can't be right, because averaging four noisy days still leaves some noise. Here's the reasoning. A lambda of 0.25 means real differences between people make up one part of the variation, and day-to-day noise makes up three parts. Averaging four days divides the noise by four, so it becomes three quarters of a part.

**Sarah:** So lambda becomes one divided by one and three quarters, which is about 0.57. And the observed slope would be two times 0.57, so about 1.1 millimetres of mercury per gram.

**Kiffer:** Right. Four recalls more than double the slope the study can see, and it's still well short of the true value. That's why repeated measures reduce regression dilution, and why a correction using a validation subsample is still worth doing.

**Sarah:** Digit preference belongs here too, and I think it's easy to underrate. Suppose readings are taken by hand and the person recording tends to round to numbers ending in zero. A true systolic reading of 138 written down as 140 crosses a common threshold for high blood pressure.

**Kiffer:** And this error behaves differently from regression dilution. It's systematic, so a larger sample just reproduces it with more precision. It does its damage near cut-points, which is why heaping at a value like twenty-five hundred grams for low birth weight matters so much. Automated devices that display the exact value remove most of it.

**Sarah:** Question five. Does an objective instrument protect a study from information bias? I think students tend to assume a device reading beats a questionnaire every time.

**Kiffer:** Often it does, but a device can carry its own differential error. A research team at the University of Michigan reported in 2020, using data from adult inpatients, that among pulse oximeter readings of ninety-two to ninety-six percent, the arterial blood test showed oxygen saturation below eighty-eight percent in 11.7 percent of measurements for Black patients, compared with 3.6 percent for White patients.

**Sarah:** So the error depends on a characteristic of the patient. If a study used pulse oximetry to define low oxygen and compared Black and White patients, it would undercount low oxygen in Black patients, and a real disparity could shrink or disappear.

**Kiffer:** That's the equity point in this lesson made concrete. The question to ask of any instrument, objective or not, is where it was developed and checked, and whether its accuracy is the same in every group the study compares.

**Sarah:** And a null finding from an instrument that can't see the difference isn't evidence that the difference is absent.

**Kiffer:** Let's pull it together with three things to take away.

**Sarah:** First, always ask whether the errors depend on the other variable. Equal errors usually pull an estimate toward the null, and unequal errors can push it either way, or create an association where none exists.

**Kiffer:** Second, the direction of a bias is an expectation, and a single study can still land on either side of the truth. Name the conditions, and check them with plausible values of sensitivity and specificity.

**Sarah:** And third, ask how the outcome was found and how the exposure was measured. Unequal detection, a single noisy reading, rounding near a threshold, or an instrument that performs differently by group can each distort a result while every recorded value looks reasonable.

**Kiffer:** If you'd like more practice, rework the birth defect recall example from this episode, keep the true odds ratio at one, and see how far the odds ratio moves if the control mothers recall the medicine only half the time.

**Sarah:** Next time, it's Lesson ten, Design-Specific and Temporal Biases, where the three kinds of bias start combining in particular study designs and in the way time is handled.

**Kiffer:** Take care, everyone.

**Sarah:** See you in Lesson ten.
