Screening &
Diagnostic Tests
Fundamental Epidemiological Concepts and Approaches
Learning objectives for this lesson:
- Distinguish screening from diagnostic uses of a test, and judge whether a test should become a screening programme using the Wilson and Jungner principles and the screening biases (lead-time bias, length-biased sampling and overdiagnosis).
- Distinguish analytic from diagnostic sensitivity, describe a test’s accuracy and precision, and calculate and interpret measures of precision and agreement (the coefficient of variation, the limits of agreement and Cohen’s kappa).
- Calculate sensitivity, specificity and the false negative and false positive fractions from a 2×2 table, interpret them as properties of the test (including a reported confidence interval and the spectrum effect), and explain and estimate the difference between true and apparent prevalence.
- Calculate predictive values from a 2×2 table, the Bayesian formulas and natural frequencies, explain why they depend on prevalence when sensitivity and specificity do not, and identify strategies that raise the positive predictive value.
- Calculate the combined sensitivity and specificity of two tests used in series or in parallel, and choose the strategy that suits the setting.
- Explain the trade-off between sensitivity and specificity as a cutpoint moves, calculate Youden’s J, interpret a ROC curve and its area under the curve (AUC), and choose a cutpoint for a given setting.
- Calculate and interpret likelihood ratios for positive, negative and category-specific results, and use them to update a pre-test probability to a post-test probability.
This course was developed by Dr. Kiffer G. Card based on Dohoo, I. R., Martin, S. W., & Stryhn, H. (2012). Methods in Epidemiologic Research. VER Inc.
Glossary: Key Terms, People & Concepts
📚 Reference page, available throughout the lesson
This glossary collects the key concepts, people, and ideas you will meet in this lesson. Use it as a reference while you work through the material, or as a review before assessments. Type in the search box to filter entries.
Introduction & Test Attributes
⏱ Estimated reading time: 12 minutes
Overview
Measures of disease frequency describe how common a condition is across a population. This lesson applies the same probabilistic vocabulary to a single test given to a single person. Whether you are evaluating a new screening assay, interpreting a clinical result or designing a surveillance algorithm, the same four-cell 2×2 logic appears: a test result that is either positive or negative, set against a true disease state that is either present or absent (Sackett & Haynes, 2002). The lesson builds up from the attributes of the test itself, through sensitivity and specificity and the predictive values that depend on prevalence, to ROC curves and likelihood ratios for tests with continuous results. This section begins by asking what a test is and why screening and diagnosis are different uses of one. It then describes the properties a test has as a measurement, which can be assessed without knowing who truly has the disease: its analytic sensitivity and specificity, its accuracy and precision, and the agreement between repeated or paired results, measured by the coefficient of variation and the limits of agreement for numbers and by Cohen’s kappa for categories.
Learning Objectives
- 1.1 Distinguish between screening tests and diagnostic tests.
- 2.1 Define analytic sensitivity and specificity of a test, and distinguish them from diagnostic sensitivity and specificity.
- 2.2 Explain the difference between accuracy and precision, and describe a test from its repeated results using the four patterns of Figure 5.1.
- 2.3 Calculate and interpret the coefficient of variation and the limits of agreement for a test that reports a number.
- 2.4 Calculate and interpret Cohen’s kappa, and explain how prevalence and bias affect it, when weighted kappa is used, and why high agreement does not show that a test is accurate.
1.1 Screening Tests and Diagnostic Tests
A test is any device or procedure designed to detect or quantify a sign, substance, tissue change or body response in an individual. Tests can also be applied at the household or other levels of aggregation. In epidemiology the term “test” extends broadly to include clinical signs, history-taking questions, survey items and post-mortem findings. The characteristics of a test matter in two contexts. In a decision-making context, such as clinical diagnosis, an appropriate test should alter your assessment of the probability that a disease exists and guide the actions that follow, such as further testing, treatment or quarantine. In a research context, understanding the characteristics of a test is essential for knowing how it affects the quality of the data.
Tests are applied for two broad purposes. Screening tests are applied to apparently healthy people to find disease early, and those who screen positive are sent on for a diagnostic work-up. Diagnostic tests are applied to people already considered abnormal, to confirm or classify their disease. Click each card below to see what each kind of test is used for, the condition that makes screening worthwhile, and examples of each.
The same test can serve either purpose, so to classify a use, look at whether the people tested have symptoms and at what happens after a positive result, rather than at the name of the test. Despite their different uses, the principles of evaluation and interpretation are the same for screening and diagnostic tests, so the rest of the lesson treats the two together, beginning with the attributes of the test itself.
2.1 Analytic Sensitivity and Specificity
Some properties of a test describe it as a measurement and can be assessed without knowing the true disease status of the people tested. The first two are laboratory properties of an assay. The analytic sensitivity of an assay is the lowest concentration of a chemical compound that it can detect. Its analytic specificity is its capacity to react to only one chemical compound. These are distinctly different from diagnostic (epidemiologic) sensitivity and specificity, which describe how well a test classifies people whose true disease status is known and are the subject of Section 2. A low detection limit does not guarantee a high diagnostic sensitivity, because people tested before the marker has formed, poor samples and reading errors still produce false negatives. When a “sensitivity” is quoted as a concentration, it is an analytic sensitivity; when it is quoted as a proportion of diseased people who test positive, it is a diagnostic sensitivity.
A test that can detect a substance must also measure it well, which raises the questions of accuracy and precision.
2.2 Accuracy and Precision
The laboratory accuracy of a test is its ability to give a true measure of the substance of interest. To be accurate, a test need not always be close to the true value, but if repeat tests are run, their average should be close to the true value. The precision of a test is the consistency of its results. If a test always gives the same value for a sample, whether or not that value is correct, it is said to be precise. Accuracy and precision are therefore separate judgements: accuracy is judged from how far the mean of repeated results lies from the true value, and precision from how widely the results scatter.
Figure 5.1 shows the four combinations as targets, in which the bullseye represents the true value and the dots represent repeated results on the same sample.
Figure 5.1. Laboratory accuracy and precision. The bullseye represents the true value.
The first target in Figure 5.1 shows a tight cluster on the bullseye (accurate and precise). The second shows a tight cluster away from it (inaccurate but precise), which is the common trap: a test whose results hardly vary can still be wrong by the same amount every time. The third shows a wide scatter centred on the bullseye (accurate but imprecise), and the fourth a wide scatter away from it (inaccurate and imprecise).
Precision is described in three settings. Repeatability is the variability obtained from repeated testing of the same sample within the same laboratory. Reproducibility is the variability obtained from testing the same sample in different laboratories. Agreement is how well two different tests, or two raters, agree when applied to the same sample. The measures used to put numbers on precision and agreement depend on whether the test reports a number or a category, beginning with tests that report a number.
2.3 Measuring Precision When a Test Reports a Number
When a test reports a number, such as a glucose concentration or a systolic blood pressure, its precision is judged by how closely repeated or paired results agree. Three measures are in common use: the coefficient of variation for repeated results on the same sample, and the concordance correlation coefficient and the limits of agreement for pairs of results from two methods. Expand each panel below to see how each measure is defined and what it reveals.
The CV describes how much repeated results on the same sample scatter, relative to their average size. It is computed as CV = σ / μ, where σ is the standard deviation among test results on the same sample and μ is their mean. Dividing by the mean matters because a scatter of 5 units is large for a measurement that averages 20 but small for one that averages 500, and the CV expresses the scatter as a fraction of the mean so that the two can be compared fairly. A lower CV indicates greater precision.
The CCC (Lin, 1989) compares two sets of test results on the same samples and better reflects agreement than a Pearson correlation. Picture the paired results plotted against each other. If the two methods agreed perfectly, every point would fall on the equality line, the 45-degree line on which both methods give the same value. The CCC asks how close the points come to that particular line, and it combines three ingredients to do so: the Pearson r, which describes how tightly the points follow some straight line; the location-shift, which describes how far the points sit above or below the equality line; and the scale-shift, which describes whether the points follow a line that is steeper or flatter than the equality line. A CCC of 1 indicates perfect agreement, and any shift or scatter pulls it below 1.
A Bland-Altman plot (Bland & Altman, 1986) looks directly at how far apart two methods are for each sample. For every pair of results, the difference between the two methods is plotted against the mean of the pair. Two kinds of line are then drawn across the plot. The mean difference (μd) shows whether one method reads higher than the other on average, which is a systematic bias. The limits of agreement (μd ± 1.96σd) mark the range within which about 95% of the differences fall. Read together, the lines and the points show how large the bias is, how far individual differences stray from it, and whether disagreement grows or shrinks with the size of the measurement.
It is tempting to judge agreement between two methods with a Pearson correlation, but a correlation does not measure agreement. It measures how closely the paired results follow a straight line, any straight line, which is association. Agreement is stricter: it asks whether the results follow the one line on which both methods give the same value. The difference shows up in two common situations. Suppose one method reads 10 units higher than the other on every sample. The points still lie on a perfect straight line, just above the equality line, so the correlation is close to 1 even though the methods never agree. Or suppose one method reads 20% higher at high values. The points again lie on a straight line, this time steeper than the equality line, and the correlation is again close to 1. A constant offset and a different slope both leave the results on a straight line, so a correlation cannot detect either of them. This is why the concordance correlation coefficient adds the location-shift and the scale-shift to the Pearson r. A Bland-Altman plot shows agreement directly. For each pair of results, the difference between the two methods is plotted on the vertical axis against the mean of the pair on the horizontal axis, and horizontal lines mark the mean difference (the systematic bias) and the 95% limits of agreement. The plot shows how large the bias is, how widely individual differences scatter around it, and whether the differences grow as the size of the measurement increases.
The coefficient of variation has no units, so it allows the precision of assays measured on different scales to be compared, but it says nothing about accuracy. The limits of agreement describe the differences to expect for individual samples, so a report should state the direction of the difference (which method minus which) and treat the mean difference as a systematic bias. Two of the measures have simple formulae. Open each tab below to work through one of them: the coefficient of variation (Equation 5.1), applied in Worked Example 5.1 to five repeated measurements of one serum sample and reproduced by Calculator 5.1, and the limits of agreement (Equation 5.2), applied in Worked Example 5.2 to two blood pressure monitors and reproduced by Calculator 5.2. In each calculator, try widening the spread of the results and watch the measure respond.
Worked Example 5.1: Coefficient of Variation
A laboratory runs the same serum sample through a glucose assay five times and obtains 102, 98, 105, 95, and 100 mg/dL. The mean of the five results is μ = 100 mg/dL, and their sample standard deviation (dividing by n − 1 = 4) is σ = 3.81 mg/dL.
CV = 3.81 / 100 = 0.038, or 3.8%. Because the CV divides by the mean, it has no units, which allows the precision of assays measured on different scales to be compared.
Worked Example 5.2: Limits of Agreement
Two automated blood pressure monitors are applied to the same 50 patients. The difference in systolic pressure (monitor A minus monitor B) has a mean of μd = 2.4 mmHg and a standard deviation of σd = 5.0 mmHg.
- Lower limit = 2.4 − 1.96 × 5.0 = 2.4 − 9.8 = −7.4 mmHg.
- Upper limit = 2.4 + 1.96 × 5.0 = 2.4 + 9.8 = 12.2 mmHg.
For about 95% of patients, monitor A is expected to read between 7.4 mmHg lower and 12.2 mmHg higher than monitor B. The mean difference of 2.4 mmHg indicates a small systematic bias, and whether a range of this width is acceptable is a clinical judgement.
Worked Example 5.1 gave a CV of 3.8%, showing that the repeated results are consistent with one another, and Worked Example 5.2 showed a small systematic bias between the monitors, with limits whose acceptability is a clinical judgement rather than a statistical one. Both measures describe tests that report a number. Many tests instead place a result in a category, such as positive or negative, and agreement between categorical results needs a measure of its own.
2.4 Agreement for Categorical Results: Cohen’s Kappa
Cohen’s kappa is the usual measure of agreement between two categorical classifications of the same items. It is defined first and applied to two radiologists, and then examined for the ways in which prevalence, bias and ordered categories affect it.
2.4.1 Agreement Beyond Chance
When test results are categorical (dichotomous or ordinal), Cohen’s kappa (κ) measures agreement beyond what would be expected by chance alone (Cohen, 1960). The reason for removing chance is that two raters, or two tests, will agree on some results even if neither is paying attention. If each rater calls about half of the results positive, they would agree on about half of them by guessing alone, much as two people tossing coins would match about half the time. A simple percentage of agreement therefore flatters the raters, because part of it would have happened anyway. Kappa corrects for this. The chance-expected agreement, pe, is worked out from how often each rater uses each category, and kappa then asks how much of the possible agreement beyond chance the raters actually achieved. It compares the observed agreement, po, with the chance agreement (Equation 5.3): the top of the fraction is the agreement the raters achieved beyond chance, and the bottom is the most agreement beyond chance that was possible.
Kappa ranges from −1 to 1. A value of 0 means agreement no better than chance, 1 means perfect agreement, and negative values mean agreement below chance, which can happen when two tests tend to pull in opposite directions. Table 5.1 gives the conventional labels of Landis and Koch (1977). They are conventions for describing kappa values, and other authors draw the bands differently, so a report gives the value itself as well as its label.
Table 5.1. Conventional labels for values of kappa (Landis and Koch, 1977).
| κ Value | Interpretation |
|---|---|
| ≤ 0 | Poor agreement |
| 0.01 – 0.20 | Slight agreement |
| 0.21 – 0.40 | Fair agreement |
| 0.41 – 0.60 | Moderate agreement |
| 0.61 – 0.80 | Substantial agreement |
| 0.81 – 1.00 | Almost perfect agreement |
Worked Example 5.3 applies Equation 5.3 to two radiologists who read the same 100 chest X-rays. Work it in three steps (po, then pe, then κ), and watch how far the chance agreement pulls kappa below the observed agreement.
Worked Example 5.3: Cohen’s Kappa
Two radiologists independently read the same 100 chest X-rays and classify each one as positive or negative for pneumonia.
| Radiologist B positive | Radiologist B negative | Total | |
|---|---|---|---|
| Radiologist A positive | 40 | 10 | 50 |
| Radiologist A negative | 5 | 45 | 50 |
| Total | 45 | 55 | 100 |
- Observed agreement: po = (40 + 45) / 100 = 0.85.
- Agreement expected by chance: radiologist A calls 50% of the films positive and radiologist B calls 45% positive, so pe = (0.50 × 0.45) + (0.50 × 0.55) = 0.225 + 0.275 = 0.50.
- κ = (0.85 − 0.50) / (1 − 0.50) = 0.35 / 0.50 = 0.70.
The radiologists agree on 85% of the films, but agreement on half of the films (pe = 0.50) would be expected by chance alone. Kappa rescales the agreement beyond chance so that it ranges from −1 to 1, with 0 for chance agreement, 1 for perfect agreement and negative values when agreement is below chance; a value of 0.70 falls in the substantial band of Table 5.1.
The radiologists agree on 85% of the films, but because half of that agreement is expected by chance, kappa is 0.70, substantial agreement in Table 5.1. Calculator 5.3 reproduces the example, so that you can see how changing the cell counts changes kappa.
The same steps work for an agreement table with more than two categories, such as ratings of good, borderline and poor. The observed agreement po is the sum of the cells on the diagonal (where the two raters chose the same category) divided by the total. The chance agreement pe adds, for each category in turn, the proportion of items that rater A placed in it multiplied by the proportion that rater B placed in it. Equation 5.3 then gives kappa as before.
Agreement Is Not Accuracy
Kappa compares two sets of readings with each other, not with the truth. High agreement shows that the readers or tests are consistent, which is precision (reliability). It does not show that either of them is accurate: two readers who make the same mistakes agree closely and are both wrong, like the tight but off-centre cluster of Figure 5.1. Whether a reading is correct can only be judged against a reference standard, through the sensitivity and specificity of Section 2.
2.4.2 How Prevalence and Bias Affect Kappa
Kappa depends on more than how well two raters agree. Because pe is computed from how often each rater uses each category, kappa also depends on the prevalence of the condition among the items rated and on any difference between the raters in how often they call a result positive (Feinstein & Cicchetti, 1990; Byrt, Bishop & Carlin, 1993). Both influences are frequent sources of error in interpreting kappa.
Factors Affecting Kappa
Prevalence: When the condition is very common or very rare, nearly every item falls in the same category for both raters, so most of the agreement sits in one cell of the table. Agreement of that kind is exactly what chance would produce, so pe is close to 1. That leaves very little room for agreement beyond chance (the bottom of Equation 5.3, 1 − pe, is small), and even a little disagreement then produces a low κ. Two tests will therefore have a higher κ when prevalence is moderate (~0.5) than when it is very high or very low, even when they make the same errors. This is the paradox of high observed agreement with a low kappa.
Bias: If one test consistently produces more positive results than the other, κ will be affected. Use McNemar’s χ² test to check whether the two tests classify the same proportion as positive before evaluating agreement. The test uses only the two cells in which the tests disagree (call their counts b and c; in Worked Example 5.3 they are 10 and 5): χ² = (b − c)² ÷ (b + c), on 1 degree of freedom. A value above 3.84 (p < 0.05) suggests that one test calls more results positive than the other; a small value gives no evidence of such a systematic difference.
A small example shows the prevalence effect. Two readers classify 100 films; reader A calls 3 positive and reader B calls 2 positive, and they never call the same film positive. They agree on the 95 films that both call negative, so po = 0.95. Chance agreement is pe = (0.03 × 0.02) + (0.97 × 0.98) = 0.951, so κ = (0.95 − 0.951) ÷ (1 − 0.951) = −0.02. Agreement of 95% is no better than chance here, because almost all of it is agreement on negatives that chance alone would produce. Since nearly every film is negative, two readers who called every film negative without looking would agree just as often. Kappa is not wrong in this case: it correctly reports that the readers never agree on which films are positive. But a reader who saw only the kappa value might wrongly conclude that the readers are careless on every film. Byrt and colleagues (1993) therefore recommended reporting a prevalence index and a bias index alongside kappa, together with the prevalence-adjusted bias-adjusted kappa, PABAK = 2po − 1. PABAK shows what kappa would be if both categories were equally common and the raters used them equally often, so it separates the effect of rarity from the raters' actual consistency. For the two readers, PABAK = 2 × 0.95 − 1 = 0.90, and the gap between 0.90 and −0.02 shows how much of the low kappa comes from the rarity of positives.
2.4.3 Weighted Kappa
Ordinary kappa treats every disagreement as equally serious. For categories with no order, such as positive and negative, that is reasonable. For tests measured on an ordinal scale, it is not: if one rater scores a film 4 and the other scores it 5, they nearly agree, whereas scores of 1 and 5 are far apart. A weighted kappa accounts for this partial agreement by giving pairs of results that are close more credit than pairs that are far apart. It does so by attaching a weight to each cell of the agreement table that reflects how serious that particular disagreement is. Each cell of the k × k agreement table, where k is the number of categories, receives a disagreement weight that is 0 on the diagonal (full agreement) and grows with the distance between the two categories assigned, i and j. Linear weights grow in equal steps with the distance, and quadratic weights grow faster, so they penalise large disagreements much more than small ones: linear weights are |i − j| ÷ (k − 1), and quadratic weights are (i − j)² ÷ (k − 1)². Weighted kappa is 1 minus the ratio of the weighted observed disagreement to the weighted disagreement expected by chance, κw = 1 − Σwijoij ÷ Σwijeij, where oij and eij are the observed and chance-expected proportions in each cell. The idea is the same as in ordinary kappa: the weighted disagreement the raters actually showed is compared with the weighted disagreement chance would produce, so a value of 1 means no disagreement at all and 0 means no better than chance. With quadratic weights it is closely related to the intraclass correlation coefficient (Fleiss & Cohen, 1973). The choice of weights changes the value, so a report states which weights were used. Weighted kappa gives a better reflection of agreement for ordinal data than the unweighted version, which treats every disagreement as equally serious.
This section has distinguished screening tests from diagnostic tests and described the attributes of a test itself: analytic sensitivity and specificity, accuracy and precision, and the coefficient of variation, the limits of agreement and kappa as measures of precision and agreement. None of these attributes shows how often a test classifies a person’s disease status correctly, which requires comparison with a gold standard and is the subject of the next section. The key takeaways and the knowledge check below review the material of this section.
Key Takeaways
- 1.1 A test is any procedure that detects or measures a sign, substance, tissue change or response in a person, and in epidemiology the word also covers clinical signs, history questions and survey items. Whether a test is being used for screening or for diagnosis depends on who is tested and why. Screening tests are applied to apparently healthy people to find disease early, before it becomes clinical, and those who screen positive are sent on for a fuller diagnostic work-up (mammography for breast cancer is an example). Diagnostic tests are applied to people already considered abnormal, to confirm or classify their disease and guide treatment (a biopsy to confirm cancer, for example). When a question asks you to classify a use, look in the scenario for whether the people tested have symptoms and what happens after a positive result, rather than at the name of the test. The principles you use to evaluate and interpret the test are the same for both uses.
- 2.1 The words sensitivity and specificity are used in two different ways in this lesson, so it is worth keeping them apart from the start. Analytic sensitivity is a laboratory property of an assay: the lowest concentration of a substance it can detect in a sample. Analytic specificity is the assay’s capacity to react to only the one compound it is meant to detect. Diagnostic (epidemiologic) sensitivity and specificity, which you calculate in the next section, describe something else: how well the test classifies people whose true disease status is known. A low detection limit does not guarantee a high diagnostic sensitivity, because people tested before the marker has formed, poor samples and reading errors still produce false negatives. So when a brochure or a question quotes a “sensitivity” as a concentration (for example in mmol/L), it is describing analytic sensitivity; when it gives the proportion of diseased people who test positive, it is describing diagnostic sensitivity.
- 2.2 Accuracy and precision describe two different qualities of repeated results on the same sample. A test is accurate when the average of its repeated results is close to the true value, even if single results scatter, and it is precise when its results are consistent with one another, whether or not they are correct. Figure 5.1 shows the four combinations as targets: accurate and precise (a tight cluster on the bullseye), inaccurate but precise (a tight cluster off the bullseye), accurate but imprecise (a wide scatter centred on the bullseye) and inaccurate and imprecise (a wide scatter away from it). When a question gives you the mean and standard deviation of repeated readings and the true value, judge accuracy from how far the mean is from the true value and precision from the size of the standard deviation, and make the two judgements separately. The usual trap is to treat consistency as correctness: a meter whose readings hardly vary can still be wrong by the same amount every time.
- 2.3 When a test reports a number, such as a glucose concentration or a blood pressure, its precision is described with numbers too. The coefficient of variation, CV = σ / μ, divides the standard deviation of repeated results on the same sample by their mean; in Worked Example 5.1, CV = 3.81 / 100 = 0.038, or 3.8%. Because the CV has no units, it lets you compare the precision of assays measured on different scales, and a lower CV means greater precision, although it tells you nothing about accuracy. The 95% limits of agreement, μd ± 1.96σd, compare two methods used on the same people: in Worked Example 5.2 they run from −7.4 to 12.2 mmHg, so for about 95% of patients monitor A reads between 7.4 mmHg lower and 12.2 mmHg higher than monitor B. When you interpret limits of agreement, state the direction of the difference (which method minus which), treat the mean difference as a systematic bias, and remember that whether the range is narrow enough is a clinical judgement, not a statistical one.
- 2.4 Cohen’s kappa (κ) measures how well two raters or two tests agree on categorical results once the agreement expected by chance alone is taken out. Work it in three steps: the observed agreement po is the sum of the agreeing (diagonal) cells divided by the total; the chance agreement pe adds, for each category, the proportion rater A placed in it multiplied by the proportion rater B placed in it; and κ = (po − pe) / (1 − pe). In Worked Example 5.3, po = 0.85, pe = 0.50 and κ = 0.70, which Table 5.1 labels substantial agreement, and a good answer reports both the value and its label. Kappa is affected by prevalence (when the condition is very common or very rare, pe is high and kappa can be low despite high observed agreement, which PABAK = 2po − 1 helps to show) and by bias (one rater calling more results positive than the other, which McNemar’s test checks). For ordered categories such as good, borderline and poor, a weighted kappa gives partial credit to near misses, and a report should say which weights were used. Finally, kappa compares two sets of readings with each other, not with the truth, so high agreement shows precision (reliability), not accuracy, which can only be judged against a reference standard.
1. A pharmacy offers a free point-of-care HbA1c test to every adult who visits during a health fair, and anyone with a result of 6.5% or higher is advised to see a physician for a fasting glucose test. Separately, a family physician orders a fasting glucose test for a patient who has lost weight and reports thirst and frequent urination. Which statement about these two uses of testing is correct?
2. A laboratory validates a new urine assay for a cannabis metabolite. Its report states that the assay detects concentrations as low as 5 ng/mL and does not react with related compounds. A clinic manager concludes: “So the assay will identify nearly every patient who has used cannabis in the past week.” Which response is correct?
3. A quality-control sample with a known total cholesterol of 5.00 mmol/L is measured ten times on each of two point-of-care meters. Meter P gives a mean of 5.02 mmol/L with a standard deviation of 0.30 mmol/L. Meter Q gives a mean of 5.60 mmol/L with a standard deviation of 0.04 mmol/L. Which description matches the patterns of Figure 5.1?
4. A wrist blood pressure monitor and a standard arm cuff are used on the same 40 adults. The differences in systolic pressure (wrist minus arm) have a mean of −3.0 mmHg and a standard deviation of 4.0 mmHg. What are the 95% limits of agreement, and what do they mean?
5. Two school nurses independently check the same 200 children for head lice, with the results below. Calculate Cohen’s kappa and choose the correct interpretation.
| Nurse B: lice | Nurse B: no lice | Total | |
|---|---|---|---|
| Nurse A: lice | 6 | 5 | 11 |
| Nurse A: no lice | 4 | 185 | 189 |
| Total | 10 | 190 | 200 |
6. A regional laboratory and a reference laboratory each read the same 150 rapid test strips. They disagree on 16 strips: the regional laboratory alone calls 14 of them positive, and the reference laboratory alone calls the other 2 positive. Using McNemar’s χ² = (b − c)² ÷ (b + c), where b and c are the two discordant counts, with 3.84 as the 5% critical value, which conclusion is correct?
✦ Pass the knowledge check with 100% to continue
Sensitivity & Specificity
⏱ Estimated reading time: 15 minutes
Overview
Section 1 described the attributes a test has as a measurement. This section turns to the two quantitative properties that capture most of what we care about in a test used on people: sensitivity, its ability to find disease that is truly present, and specificity, its ability to say “no” correctly when disease is truly absent. Both are judged against a gold standard and calculated from a 2×2 table of test result by true disease status. Because each is calculated within one disease group, they are properties of the test, although they can still shift with the kind of disease in the people tested. The section ends by using sensitivity and specificity to move between the true prevalence of a condition and the prevalence a test reports, and by applying all of these measures to the validation of an administrative case definition.
Learning Objectives
- 3.1 Explain the concept of a gold standard and its role in test evaluation.
- 3.2 Calculate sensitivity, specificity, false positive fraction, and false negative fraction from a 2×2 table, and interpret a confidence interval reported for sensitivity or specificity.
- 3.3 Explain why sensitivity and specificity are properties of the test, and why they may still differ between settings (the spectrum effect).
- 3.4 Distinguish between true prevalence and apparent prevalence.
- 3.5 Estimate true prevalence from apparent prevalence using the Rogan-Gladen formula.
- 3.6 Apply sensitivity, specificity, predictive value and the Rogan-Gladen correction to the validation of an administrative case definition.
3.1 The Gold Standard
Sensitivity and specificity are estimated by comparing the results of a test with a reference that is taken to give the true disease status. This kind of comparison was met earlier, when questionnaires were validated in Lesson 3; the recall below links the two, and its retrieval question checks the distinction between reliability and validity evidence.
Recall: Criterion Validity
HSCI 341 Lesson 3, Section 3 (Wording, Structure and Pre-Testing), listed four approaches to validating a questionnaire: comparison with a gold standard, comparison with established methods, repeated administration and method comparison. Comparison with a gold standard gives criterion validity evidence, and sensitivity and specificity, the subject of this section, are that evidence for a categorical measure. Repeated administration to the same respondents is a test-retest design, so it gives reliability evidence: it shows that answers are repeatable, and a repeatable instrument can still measure the wrong thing.
Retrieval question. Classify each of the four Lesson 3 approaches as reliability evidence or validity evidence.
Comparison with a gold standard gives criterion validity evidence. Comparison with a well-validated established instrument gives validity evidence (concurrent or convergent), which is only as strong as the established instrument. Repeated administration gives test-retest reliability evidence. Method comparison, which checks whether mail and telephone administration give comparable results, shows agreement between modes, which is reliability (equivalence) evidence unless one mode is treated as the reference.
A gold standard (GS) is a test or procedure that is absolutely accurate: it diagnoses all cases of a specific disease and misdiagnoses none. In reality, very few true gold standards exist. Much of the difficulty comes from biological variability. People do not switch from healthy to “diseased” at the moment of exposure; the disease develops over time, and the time it takes to cross a threshold that any test can detect varies from person to person. Two people infected on the same day may therefore look different on the same test a week later, and even the best reference can miss someone early in the course of the disease. This matters because the estimates of sensitivity and specificity can only be as good as the reference against which they are calculated. Where the reference misclassifies some people, the new test is judged against the wrong answer for them. A test that correctly detects a case the reference missed is then counted as a false positive, so the error of the reference is wrongly charged to the test.
When No Gold Standard Exists
When no true gold standard exists, alternative approaches for estimating sensitivity and specificity are needed, including the use of results from several different tests, repeated testing of selected samples, and latent class models (Hui and Walter, 1980). Latent class models estimate sensitivity and specificity by treating true disease status as an unobserved category, inferred from the pattern of results when several imperfect tests are applied to the same people.
With a gold standard, or the best available reference, in place, each person tested can be classified twice, once by the test and once by the reference, and these two classifications are arranged in a 2×2 table.
3.2 Calculating Sensitivity and Specificity
Sensitivity and specificity, and their complements, the false negative and false positive fractions, are all calculated from the 2×2 table of test result by true disease status. The table is set out first, then the four measures are calculated from it, and finally the confidence intervals that accompany them are interpreted.
3.2.1 The 2×2 Table
Table 5.2 gives the notation used throughout the lesson. Its rows give the true disease status and its columns the test result. Cells a and d hold the correct results and cells b and c the errors; m1 and m0 are the numbers with and without the disease, and n1 and n0 are the numbers who test positive and negative.
Table 5.2. The 2×2 table of disease status by test result, with the notation for its cells and totals.
| Test Positive (T+) | Test Negative (T−) | Total | |
|---|---|---|---|
| Disease Positive (D+) | a (true positive) | b (false negative) | m1 |
| Disease Negative (D−) | c (false positive) | d (true negative) | m0 |
| Total | n1 | n0 | n |
Many published tables, including the worked examples in this lesson, put disease status in the columns instead, and Figure 5.2 shows that layout together with the direction in which each measure is read. The letters keep their meaning (a is always the true positives, b the false negatives, c the false positives and d the true negatives), so the safe habit is to find the disease-status totals first, whichever way the table is drawn.
3.2.2 Sensitivity, Specificity and the Error Fractions
Sensitivity (Se) is the proportion of truly diseased people who test positive, and specificity (Sp) is the proportion of truly non-diseased people who test negative. Their complements are the error fractions: the false negative fraction, 1 − Se, is the share of diseased people the test misses, and the false positive fraction, 1 − Sp, is the share of non-diseased people it wrongly calls positive. Click each card below to see the formula for each measure in the notation of Table 5.2, when each kind of error matters most, and the mnemonics for using sensitive and specific tests.
Equation 5.4 and Equation 5.5 give sensitivity and specificity in terms of the cells of the 2×2 table. In each, the denominator is a disease-status total, everyone with the disease for Se and everyone without it for Sp.
Worked Example 5.4 applies Equations 5.4 and 5.5 to a study of 188 stool samples tested for norovirus with an enzyme immunoassay (EIA). As you read it, check that each measure is divided by a column total of the reference standard, not by a row total of test results.
Worked Example 5.4: Norovirus EIA Data
From a study of 188 stool samples from outbreaks in Ontario, tested with an EIA against a reference standard built from several tests, which this example treats as the gold standard (data from Fisman et al., 2009):
| GS+ (D+) | GS− (D−) | Total | |
|---|---|---|---|
| T+ | 71 | 3 | 74 |
| T− | 11 | 103 | 114 |
| Total | 82 | 106 | 188 |
- Se = 71/82 = 86.6% (95% CI: 77.3%, 93.1%)
- Sp = 103/106 = 97.2% (95% CI: 92.0%, 99.4%)
- FNF = 1 − 0.866 = 13.4%
- FPF = 1 − 0.972 = 2.8%
The sensitivity is 71/82, not 71/74: the denominator is the 82 samples with norovirus, not the 74 that tested positive. Dividing by the number who tested positive gives the predictive value of Section 3 instead, which is the most common error in these calculations. Calculator 5.4 reproduces the example, so that you can see how changing the cell counts changes both measures.
3.2.3 Reading a Confidence Interval
Sensitivity and specificity estimated from a study sample are estimates of the test’s true values, and a report gives each with a 95% confidence interval, as in Worked Example 5.4. The interval is the range of values for the true sensitivity that are consistent with the data: if the study were repeated many times, 95% of the intervals calculated in this way would contain the true value. In plain words, the norovirus EIA could detect as few as about 77 or as many as about 93 of every 100 infected people. The width of an interval depends mainly on how many people are in its denominator. Sensitivity is estimated only from the 82 people with the disease, and specificity from the 106 without it, so the sensitivity interval is the wider of the two; a study that enrolled more people with the disease would narrow it. This lesson does not ask you to calculate them.
Because each measure is calculated within one disease group, the next question is what that implies about carrying sensitivity and specificity from one setting to another.
3.3 Sensitivity and Specificity as Properties of the Test
Sensitivity and specificity are calculated within the diseased group and within the non-diseased group, down the columns of Figure 5.2. Sensitivity looks only at people who have the disease and asks what share of them the test detects; specificity looks only at people who do not and asks what share of them the test correctly clears. Neither calculation uses information about how many people are in the other group, so the prevalence of the condition does not enter either of them. If a population had twice as many cases, the sensitivity would stay the same, because twice as many cases would be detected out of twice as many diseased people. For this reason sensitivity and specificity are called properties of the test, and they can be carried from one setting to another more readily than predictive values, which are read across the rows, mix the two groups together and therefore depend on prevalence (Section 3). The same reasoning explains why a highly sensitive test is used to rule disease out after a negative result, and a highly specific test to rule it in after a positive result, as the SnNOut and SpPIn cards in 3.2.2 summarise.
They still depend on who was tested. A test evaluated in people with advanced or severe disease often looks more sensitive than it will be in people with early or mild disease, who may have lower levels of the marker the test detects; this is the spectrum effect. For example, an antibody test evaluated in patients with long-standing infection may miss people tested in the first weeks after infection, before antibodies have formed, so a negative result early in an illness does not rule the disease out. Values estimated in one setting, such as a hospital study, may therefore not hold in a screening population, and before using a published sensitivity you should check whether the people in the evaluation resemble the people you will test.
Design Biases in Diagnostic Accuracy Studies
Published sensitivity and specificity can be inflated by the way a study is designed (Lijmer et al., 1999). Comparing people with severe disease against healthy volunteers, instead of testing the kind of people the test is meant for, produces the spectrum effect described above. Partial verification occurs when only some people, often those who test positive, receive the reference standard, and differential verification when people receive different reference standards depending on their test result. Reading the test while knowing the reference result, or the reverse, also makes the two agree more often. QUADAS-2 (Whiting et al., 2011) is a tool for assessing the risk of bias and the applicability of a diagnostic accuracy study in four domains: patient selection, the index test, the reference standard, and flow and timing. STARD 2015 (Bossuyt et al., 2015) is a reporting checklist that sets out what a diagnostic accuracy study should report, so that readers can judge these problems for themselves.
Sensitivity and specificity describe how a test performs within the diseased and the non-diseased groups. When a test is applied to a whole population, these two quantities and the prevalence together determine the proportion that tests positive.
3.4 True and Apparent Prevalence
A survey that uses an imperfect test counts the people who test positive, which is generally a different number from the people who have the condition. The recall below restates the definition of prevalence from Lesson 4 and its link with incidence and duration, and its retrieval question gives practice with that link.
Recall: Prevalence
Prevalence is the proportion of a population that has a condition at a point in time (point prevalence) or during a period (period prevalence). HSCI 341 Lesson 4, Section 3 (Prevalence, Mortality and Other Frequency Measures), showed that it reflects both how often new cases arise and how long they last: in a steady state, P ÷ (1 − P) = I × D, so P ≈ I × D when the condition is uncommon. HSCI 230 Lesson 3, Section 4, called this the prevalence-duration confound, because factors that prolong a condition can look like causes of it in a cross-sectional study.
Retrieval question. A condition has an incidence rate of 2 per 1,000 person-years and an average duration of 10 years. What is its approximate prevalence in a steady state?
I × D = 0.002 × 10 = 0.02, so the prevalence odds are 0.02 and the prevalence is 0.02 ÷ 1.02 = 0.0196, about 2%.
The true prevalence (P) is the actual proportion of the population that has the disease. In Worked Example 5.4, P = 82/188 = 43.6%. The apparent prevalence (AP) is the proportion that tests positive, which includes the false positives and leaves out the false negatives. In Worked Example 5.4, AP = 74/188 = 39.4%. Equation 5.6 expresses the apparent prevalence in terms of the true prevalence and the sensitivity and specificity of the test. Its first term is the true positives, the diseased share of the population multiplied by the sensitivity, and its second term is the false positives, the healthy share multiplied by the false positive fraction.
Worked Example 5.5 applies Equation 5.6 to the norovirus study. It shows that the formula recovers the apparent prevalence counted from the table, and it separates that figure into its true positive and false positive parts.
Worked Example 5.5: Apparent Prevalence from the Formula
In the norovirus study, the true prevalence is P = 82/188 = 0.4362, and the test has Se = 0.8659 and Sp = 0.9717.
AP = (0.4362 × 0.8659) + (1 − 0.4362) × (1 − 0.9717) = 0.3777 + (0.5638 × 0.0283) = 0.3777 + 0.0160 = 0.3937, or 39.4%.
The formula reproduces the 74/188 = 39.4% counted directly from the table. The first term is the share of the population that is diseased and correctly detected (the true positives), and the second is the share that is healthy but tests positive (the false positives).
Whether the apparent prevalence is too high or too low depends on a tug of war between two kinds of error. The missed cases (false negatives) pull the apparent prevalence down, because people who have the disease are not counted. The false positives push it up, because people without the disease are counted. Here the 11 missed cases outnumber the 3 false positives, so the apparent prevalence (39.4%) is below the true prevalence (43.6%). When a condition is uncommon the balance usually reverses. The healthy group is then very large, and even a small false positive fraction applied to a large group produces many false positives, while the small diseased group can produce only a few missed cases. The false positives then win the tug of war and push the apparent prevalence above the true prevalence. For that reason the proportion testing positive should be called a positivity rate, not a prevalence, unless it has been corrected. Calculator 5.5 reproduces the example; try lowering the true prevalence and watch the apparent prevalence overtake it.
Equation 5.6 runs from the true prevalence to the apparent prevalence. A survey usually needs the reverse.
3.5 Estimating True Prevalence: The Rogan-Gladen Formula
If the Se and Sp of a test are known, the true prevalence can be estimated from the apparent prevalence using the Rogan-Gladen formula (Rogan & Gladen, 1978), Equation 5.7, which is Equation 5.6 solved for P. The logic can be read from its parts. The numerator, AP + Sp − 1, is the apparent prevalence with the false positive fraction (1 − Sp) taken away, which removes the people counted only because the test wrongly called them positive. The denominator, Se + Sp − 1, scales the result back up to allow for the cases the test misses. A perfect test has Se + Sp − 1 = 1 and needs no correction, while a test no better than a coin toss has a denominator of 0 and cannot be corrected at all. Keep every value as a proportion rather than a percentage so that the “− 1” terms make sense, and work out the numerator and the denominator separately before dividing.
Worked Example 5.6 applies Equation 5.7 to an apparent prevalence of 0.150 measured with a test whose sensitivity is 0.363 and specificity 0.876. Note the check at its end on whether the estimate lies between 0 and 1.
Worked Example 5.6: True Prevalence from the Rogan-Gladen Formula
If AP = 0.150, Se = 0.363, and Sp = 0.876, then:
P = (0.150 + 0.876 − 1) / (0.363 + 0.876 − 1) = 0.026 / 0.239 = 0.109 (10.9%)
Note: Some combinations of Se, Sp, and AP can produce estimates of P outside the range 0–1, indicating that the Se and Sp estimates may not be applicable to the population being studied.
The corrected estimate (10.9%) is lower than the apparent prevalence (15.0%) because, with a specificity of 0.876, the false positives outnumber the cases the test misses; when missed cases outnumber false positives the correction runs the other way. Calculator 5.6 reproduces the example, so that you can see how different values of Se and Sp change the corrected estimate.
These measures, and the Rogan-Gladen correction, apply to any rule that classifies people as cases or non-cases, including a case definition applied to health records.
3.6 Validating an Administrative Case Definition
Surveillance in Canada often counts cases from health administrative records rather than by testing people. The recall below returns to the Canadian Chronic Disease Surveillance System from Lesson 2 and explains why its case definitions can be evaluated as tests.
Recall: The Canadian Chronic Disease Surveillance System
HSCI 341 Lesson 2, Section 1 (Surveillance Systems and Canadian Data Sources), introduced the Canadian Chronic Disease Surveillance System (CCDSS), which applies validated case definitions to health administrative records to estimate the prevalence and incidence of chronic conditions such as diabetes, hypertension and dementia. A case definition of this kind works as a test: each person is classified as a case or a non-case from their records, and the classification can be checked against a reference standard such as chart review.
Retrieval question. Which two kinds of administrative record does the CCDSS draw on?
Physician billing claims and hospital discharge abstracts.
Because a case definition works as a test, its validation uses the same 2×2 table, with the chart result as the reference, and the same measures. Worked Example 5.7 validates a diabetes case definition against chart review and then applies the Rogan-Gladen formula (Equation 5.7) to a provincial prevalence estimate. Watch for the step at which the false positives among the large group without diabetes push the administrative estimate above the truth.
Worked Example 5.7: Validating an Administrative Case Definition
A diabetes case definition used with Canadian administrative data counts a person as a case if they have one hospital discharge abstract, or two physician claims within two years, with a diabetes diagnosis. To validate it, a study abstracts the charts of 1,000 adults and compares the case definition with diabetes as documented in the chart, the reference standard. The counts are hypothetical, chosen for teaching.
| Chart: diabetes | Chart: no diabetes | Total | |
|---|---|---|---|
| Case definition met | 86 | 18 | 104 |
| Case definition not met | 14 | 882 | 896 |
| Total | 100 | 900 | 1,000 |
- Se = 86/100 = 0.86
- Sp = 882/900 = 0.98
- PV+ = 86/104 = 0.83
The definition misses 14% of people with diabetes and wrongly includes 2% of people without it, and 17% of the people it counts as cases do not have diabetes according to the chart. The predictive value depends on the prevalence of diabetes in the validation sample (10% here), so it would be lower where diabetes is less common, as the next section shows.
These values change a prevalence estimate. Suppose the case definition, applied to a province's administrative data, gives an apparent prevalence of 8.0%. The Rogan-Gladen formula gives P = (0.080 + 0.98 − 1) ÷ (0.86 + 0.98 − 1) = 0.060 ÷ 0.84 = 0.071, or 7.1%. The administrative estimate is too high by about 0.9 percentage points, because the false positives among the 92.9% of people without diabetes (0.02 × 0.929, or 1.9 percentage points) outnumber the cases the definition misses (0.14 × 0.071, or 1.0 percentage point). A lower specificity would widen the gap, which is why validation studies of administrative case definitions report specificity and predictive value as well as sensitivity.
The example shows the full sequence for a question of this kind: set out the table with the chart as the reference, calculate the measures and put each into plain words, and then explain which kind of error, false positives or missed cases, moves the administrative estimate away from the truth.
This section has defined the gold standard, computed sensitivity and specificity from the 2×2 table, and used them to move between true and apparent prevalence. Worked Example 5.7 also computed a predictive value, which depends on the prevalence in the population tested; the next section develops predictive values in full. The key takeaways and knowledge check below review this material.
Key Takeaways
- 3.1 Sensitivity and specificity can only be measured by comparing a test with a reference that tells you who truly has the disease, and that reference is called the gold standard. A true gold standard would diagnose every case and misdiagnose none, but very few exist in practice, partly because people do not become detectably diseased the moment they are exposed and the time this takes varies from person to person. When no true gold standard exists, sensitivity and specificity are estimated in other ways, such as combining the results of several tests, retesting selected samples or using latent class models. In a question, identify the reference first (in Worked Example 5.7, for example, it is chart review) and build the 2×2 table around it, because every measure in this section is calculated against it. Keep in mind that the estimates can only be as good as the reference: where the reference itself misclassifies some people, the test is judged against the wrong answer for them.
- 3.2 Sensitivity is the proportion of people who truly have the disease whom the test calls positive, Se = a / (a + b), and specificity is the proportion of people without the disease whom the test calls negative, Sp = d / (c + d). Their complements are the error fractions: the false negative fraction, 1 − Se, is the share of diseased people the test misses, and the false positive fraction, 1 − Sp, is the share of healthy people it wrongly calls positive. Before you divide, find the reference-standard totals, because the denominator of Se is everyone with the disease and the denominator of Sp is everyone without it, whether the table puts disease status in its rows (Table 5.2) or in its columns (Worked Example 5.4). Dividing by the number who tested positive instead (71/74 rather than 71/82 in the norovirus study) gives the PV+, not the sensitivity, and is the most common mistake. When a confidence interval is reported, read it as the range of true values consistent with the data (the norovirus EIA could detect as few as about 77 or as many as about 93 of every 100 infected people), and expect it to be wider for whichever measure has fewer people in its denominator, here the 82 with the disease rather than the 106 without it.
- 3.3 Sensitivity and specificity are called properties of the test because each is calculated within one disease group (the diseased for Se, the non-diseased for Sp), so the prevalence of the condition does not enter either one and they carry from one setting to another more readily than predictive values. They still depend on who was tested: a test evaluated in people with advanced or severe disease often looks more sensitive than it will be in people with early or mild disease, which is called the spectrum effect. For example, an antibody test evaluated in long-standing infections may miss people tested in the first weeks, before antibodies form, so a negative result early in an illness does not rule the disease out. Two mnemonics tell you how to use each property: SnNOut (a highly sensitive test with a Negative result rules Out disease) and SpPIn (a highly specific test with a Positive result rules In disease). When a question asks whether published values will hold in a new setting, compare the people in the evaluation (for example a hospital study) with the people you will test (for example a screening population), and name the spectrum effect if their mix of disease differs.
- 3.4 True prevalence (P) is the proportion of a population that actually has the condition, while apparent prevalence (AP) is the proportion that tests positive. The two differ because the count of positives includes the false positives and leaves out the false negatives: in Worked Example 5.4, P = 82/188 = 43.6% but AP = 74/188 = 39.4%. Equation 5.6, AP = P × Se + (1 − P)(1 − Sp), shows where each part comes from: the first term is the true positives and the second is the false positives among the healthy. When the condition is uncommon, even a small false positive fraction applied to the large healthy group can push the apparent prevalence above the true prevalence, as in the serosurvey question in this section's knowledge check (5.7% against 5%). So in your answers, call the proportion testing positive a positivity rate, not a prevalence, unless it has been corrected for the test’s sensitivity and specificity.
- 3.5 If you know a test’s sensitivity and specificity, you can work backwards from the apparent prevalence to an estimate of the true prevalence with the Rogan-Gladen formula, P = (AP + Sp − 1) / (Se + Sp − 1). In Worked Example 5.6, an apparent prevalence of 0.150 with Se = 0.363 and Sp = 0.876 gives P = 0.026 / 0.239 = 0.109, or 10.9%. Work out the numerator and the denominator separately before dividing, and keep every value as a proportion rather than a percentage so that the “− 1” terms make sense. Check that your answer lies between 0 and 1: an estimate outside that range tells you that the Se and Sp you used probably do not apply to the population being studied. Then explain the direction of the correction: the true prevalence is lower than the apparent prevalence when false positives outnumber missed cases, and higher when missed cases outnumber false positives.
- 3.6 A case definition applied to health administrative records, such as the diabetes definition in Worked Example 5.7 (one hospital discharge abstract, or two physician claims within two years), works like a test: it classifies each person as a case or a non-case, and that classification can be checked against a reference standard such as chart review. Its validation therefore uses the same 2×2 table and the same measures: in the worked example, Se = 86/100 = 0.86, Sp = 882/900 = 0.98 and PV+ = 86/104 = 0.83. Put each into plain words: the definition misses 14% of people with diabetes, wrongly includes 2% of people without it, and 17% of the people it counts as cases do not have diabetes according to the chart. Remember that the PV+ depends on the prevalence in the validation sample, so it would be lower where diabetes is less common, while the sensitivity and specificity can be used in the Rogan-Gladen formula to correct a prevalence estimate from administrative data (8.0% corrected to 7.1% in the example). In questions of this kind, set out the table with the chart result as the reference, calculate the measures, and then explain which kind of error, false positives or missed cases, moves the administrative estimate away from the truth.
1. A study evaluates a rapid malaria test in returning travellers, using microscopy of a blood film, read by one technologist, as the reference. Microscopy is known to miss some infections in which there are few parasites in the blood. Which statement is correct?
2. A rapid test for group A streptococcus (strep throat) is compared with throat culture, the reference standard, in 300 children with a sore throat. The columns give the culture result and the rows the rapid test result. Calculate the sensitivity, specificity, false negative fraction (FNF) and false positive fraction (FPF).
| Culture: strep | Culture: no strep | Total | |
|---|---|---|---|
| Rapid test positive | 85 | 12 | 97 |
| Rapid test negative | 15 | 188 | 203 |
| Total | 100 | 200 | 300 |
3. The report on the rapid strep test (100 children with strep and 200 without, on throat culture) gives Se = 0.85 (95% CI 0.76 to 0.91) and Sp = 0.94 (95% CI 0.90 to 0.97). Which statement interprets the interval for sensitivity correctly and explains why it is the wider of the two?
4. A rapid influenza antigen test had a sensitivity of 0.90 when it was evaluated in adults admitted to hospital with severe influenza. A community clinic plans to use it for adults with one day of mild symptoms. A colleague says: “Sensitivity is a property of the test, so it will be 0.90 in the clinic too.” Which response is best?
5. In the rapid strep test study, 100 of the 300 children had strep on throat culture. The rapid test was positive in 97 children: 85 with strep (true positives) and 12 without (false positives). It was negative in 15 children who had strep (false negatives). What are the true prevalence and the apparent prevalence, and why do they differ?
6. A provincial team validates an asthma case definition for physician billing claims against chart review and finds a sensitivity of 0.75 and a specificity of 0.97. In the province’s claims data, 6.0% of adults meet the definition. Using the Rogan-Gladen formula, P = (AP + Sp − 1) ÷ (Se + Sp − 1), estimate the true prevalence and say why it differs from 6.0%.
7. Before a serosurvey, a rapid antibody test for SARS-CoV-2 is evaluated against PCR-confirmed infection and found to have Se = 0.75 and Sp = 0.98. In the serosurvey, 5.65% of adults test positive, and a dashboard reports this as “the proportion of adults who have been infected”. Using the Rogan-Gladen formula, P = (AP + Sp − 1) ÷ (Se + Sp − 1), which statement is correct?
8. For the same rapid antibody test, the evaluation reports Se = 0.75 with a 95% confidence interval of 0.68 to 0.81. Most of the infected adults in the evaluation had been admitted to hospital, whereas most infections in the serosurvey population were mild. Which statement is correct?
✦ Pass the knowledge check with 100% to continue
Predictive Values
⏱ Estimated reading time: 12 minutes
Overview
Section 2 covered sensitivity and specificity, which are properties of the test itself. This section introduces the predictive values, which tell an individual person what to believe about their disease status given the test result. Predictive values depend on the prevalence of disease in the population tested, which is why the same test can be useful in one setting and misleading in another, and this makes the section the most clinically important in the lesson. It defines the two predictive values, calculates them in three equivalent ways, shows how strongly they depend on prevalence, and describes how the predictive value of a positive result can be raised, including by combining two tests in series or in parallel.
Learning Objectives
- 4.1 Define predictive value positive (PV+) and predictive value negative (PV−), and explain how each differs from sensitivity and specificity.
- 4.2 Calculate PV+ and PV− from a 2×2 table, using Bayesian formulas and by counting natural frequencies.
- 4.3 Explain how prevalence affects predictive values.
- 4.4 Describe strategies for increasing the predictive value of a positive test.
- 5.1 Calculate the combined sensitivity and specificity of two tests used in series and in parallel, and choose between the two strategies according to the costs of false positives and false negatives.
4.1 What Predictive Values Tell You
While Se and Sp are characteristics of the test, predictive values tell us how useful the test is for individuals whose disease status is unknown. Once we decide to use a test, we want to know the probability that the individual has, or does not have, the disease given the test result. The predictive value positive (PV+) is the probability that an individual who tests positive actually has the disease, p(D+|T+). The predictive value negative (PV−) is the probability that an individual who tests negative truly does not have the disease, p(D−|T−).
The conditional probabilities run in opposite directions. Sensitivity, p(T+|D+), starts from people known to have the disease and asks how often the test is positive, which is the question from the test’s point of view. PV+, p(D+|T+), starts from people who tested positive and asks how many of them have the disease, which is the question a patient holding the result asks. In the 2×2 table of Figure 5.2, sensitivity and specificity are read down the disease-status columns, while predictive values are read across the test-result rows.
Name the Denominator
The surest way to keep the four measures apart is to name the group in the denominator: everyone with the disease for Se, everyone without it for Sp, everyone who tested positive for PV+ and everyone who tested negative for PV−. When asked what a measure means in plain words, start the sentence from that group.
Because the rows mix diseased and non-diseased people in proportion to the prevalence, predictive values also depend on how common the disease is. Before turning to that dependence, the next step is to calculate them.
4.2 Calculating Predictive Values
Predictive values can be calculated in three ways, which all give the same answer: by dividing cells of a 2×2 table, from a Bayesian formula when only Se, Sp and the prevalence are known, and by counting natural frequencies in a round number of people.
4.2.1 From the 2×2 Table and the Bayesian Formulas
From the 2×2 table, PV+ is the true positives divided by everyone who tested positive, a / n1, and PV− is the true negatives divided by everyone who tested negative, d / n0. Often, however, no 2×2 table is available for the population of interest. A clinician may know only the sensitivity and specificity reported for a test and an estimate of how common the disease is in the patients being tested. Bayes’s theorem gives the same quantities from these three numbers. The formulas look complicated, but they rebuild the 2×2 table in proportions rather than counts. In Equation 5.8 the numerator, prevalence times sensitivity, is the share of all people tested who are true positives. The denominator adds the share who are false positives, the healthy share times the false positive fraction, so that it equals everyone who tests positive. Dividing one by the other is therefore the same as dividing a by n1, and Equation 5.9 does the same for the negatives. Open each tab below to work through one predictive value: each gives the count from the norovirus study, the formula, a worked example that checks the formula against the count (Worked Examples 5.8 and 5.9), and a calculator (Calculators 5.7 and 5.8) in which you can change the prevalence and watch the predictive value respond.
Predictive Value Positive (PV+)
The PV+ is the probability that an individual who tests positive actually has the disease: p(D+|T+) = a / n1.
In the norovirus example: PV+ = 71/74 = 95.9% (95% CI: 88.6%, 99.2%)
Worked Example 5.8: PV+ from the Formula
The same answer follows from the formula, using the norovirus study’s prevalence p(D+) = 0.436, Se = 0.866, and Sp = 0.972, so that p(D−) = 1 − 0.436 = 0.564.
PV+ = (0.436 × 0.866) / [(0.436 × 0.866) + 0.564 × (1 − 0.972)] = 0.3776 / (0.3776 + 0.0158) = 0.3776 / 0.3934 = 0.960, or 96.0%.
The numerator is the share of all people tested who are true positives, and the denominator is the share who test positive for any reason. The small difference from 95.9% reflects rounding of the inputs.
Predictive Value Negative (PV−)
The PV− is the probability that an individual who tests negative truly does not have the disease: p(D−|T−) = d / n0.
In the norovirus example: PV− = 103/114 = 90.4% (95% CI: 83.4%, 95.1%)
Worked Example 5.9: PV− from the Formula
With p(D−) = 0.564, Sp = 0.972, p(D+) = 0.436, and Se = 0.866:
PV− = (0.564 × 0.972) / [(0.564 × 0.972) + 0.436 × (1 − 0.866)] = 0.5482 / (0.5482 + 0.0584) = 0.5482 / 0.6066 = 0.904, or 90.4%.
The numerator is the share of all people tested who are true negatives, and the denominator is the share who test negative for any reason, including the diseased people the test misses.
In the norovirus study both predictive values were high, 95.9% for a positive result and 90.4% for a negative result, and the formulas reproduced the counts. Both, however, were computed at the prevalence in that study, 43.6%.
4.2.2 Counting Natural Frequencies
The formulas can feel abstract, and a reliable check is to walk a round number of people through the test and count them. Split the group by the prevalence, apply the sensitivity to the diseased and the false positive fraction to the healthy, and then divide the true positives by everyone who tested positive. Worked Example 5.10 follows 10,000 people through a test with 99% sensitivity and 95% specificity at a prevalence of 1%. Write down the true positive and false positive counts before you divide, and compare them.
Worked Example 5.10: Predictive Value in Natural Frequencies
The formula can feel abstract, so it helps to walk a whole group of people through the test and simply count. Imagine screening 10,000 people for a disease with a prevalence of 1%, using a strong test with Se = 99% and Sp = 95%.
| Group | People |
|---|---|
| Have the disease (1% of 10,000) | 100 |
| Diseased and test positive (true positives, 99% of 100) | 99 |
| Diseased and test negative (false negatives) | 1 |
| Do not have the disease (99% of 10,000) | 9,900 |
| Healthy and test positive (false positives, 5% of 9,900) | 495 |
| Healthy and test negative (true negatives) | 9,405 |
Now read the positive predictive value straight off the counts. A total of 99 + 495 = 594 people test positive, but only 99 of them truly have the disease, so PV+ = 99 / 594 = 16.7%. About five of every six positive results are false alarms, even though the test is correct 99% of the time in the sick and 95% of the time in the healthy. The reason is arithmetic, not a flaw in the test: the 9,900 healthy people are so numerous that their small 5% error rate yields more false positives (495) than there are true cases in the whole group (100).
Figure 5.3 tells the same story as an interactive animation. Click through its scenes to follow 1,000 people through a test with 95% sensitivity and 95% specificity at a prevalence of 1%, and watch the positive predictive value emerge from the counts of true and false positives.
Watch a 95-95 test scan 1,000 people and see PPV emerge from the math. Next ▶ advances scenes.
Figure 5.3. A 6-scene Bayesian-reasoning visualization: a population of 1,000 with 1% prevalence, a 95%-sensitive 95%-specific test scanning across, the four buckets (TP/FP/FN/TN) populating in real time, and the surprising PPV that follows.
In both Worked Example 5.10 and Figure 5.3, a good test gives a low PV+ because the disease is rare, which leads directly to the role of prevalence.
4.3 How Prevalence Changes Predictive Values
Predictive values depend heavily on the prevalence of disease in the population tested, an application of Bayes's theorem. This is why PV+ and PV− are not good measures of a test’s intrinsic performance: they vary from population to population. Table 5.3 holds the sensitivity (86.6%) and specificity (97.2%) of the norovirus EIA fixed and recomputes both predictive values with Equations 5.8 and 5.9 at three levels of prevalence.
Table 5.3. Predictive values of the norovirus EIA (Se = 86.6%, Sp = 97.2%) at three levels of prevalence.
| Prevalence (%) | PV+ (%) | PV− (%) |
|---|---|---|
| 50 | 96.9 | 87.9 |
| 5 | 61.9 | 99.3 |
| 0.1 | 3.0 | 100.0 |
When prevalence drops to 0.1%, the PV+ falls to just 3%, meaning that 97% of positive results are false positives, while the PV− approaches 100%. The reason is arithmetic, not a fault in the test: when the healthy group is very large, even a small false positive fraction produces more false positives than there are true cases, as the counts of Worked Example 5.10 showed. The PV− moves the other way for the mirror-image reason. When the disease is rare, almost everyone who tests negative is healthy simply because almost everyone is healthy, and the few missed cases make up a tiny share of all the negative results. This is a fundamental challenge in screening low-prevalence populations: a negative result is very reassuring, but a positive result is more often wrong than right. Two practical lessons follow. First, when two settings with different prevalence give different PV+ values, the difference should be explained by prevalence, not by a change in the test, because the sensitivity and specificity are the same in both. Second, for the same reason, a predictive value measured in one population should not be carried over to another population with a different prevalence; it has to be recalculated with Equations 5.8 and 5.9 using the new prevalence.
Interactive 5.1 brings the cutoff, the prevalence and the overlap between the test scores of healthy and diseased people together in one simulator. Lower the prevalence slider and notice that sensitivity and specificity stay fixed while PV+ falls and PV− rises; then try the presets for population screening and a symptomatic clinic. Moving the cutoff trades sensitivity against specificity, which Section 4 develops.
🧪 Interactive 5.1: Sensitivity, Specificity, PPV & the Cutoff
Distribution of test scores
Drag the dashed cutoff line. Right of the line = test positive.
2×2 confusion matrix (per 10,000 tested)
| D+ | D− | Total | |
|---|---|---|---|
| T+ | 1954 | 2020 | 3974 |
| T− | 46 | 5980 | 6026 |
| Total | 2000 | 8000 | 10,000 |
If a low prevalence is what drives a low PV+, the remedies follow directly.
4.4 Raising the Predictive Value of a Positive Test
Because a low PV+ comes from false positives among the many healthy people tested, it can be raised by testing a group in which the disease is more common or by reducing the false positives. Three strategies follow from this: targeting high-risk groups, increasing specificity and using more than one test. Click each card below to see how each strategy raises PV+, with an example and the trade-off it involves.
Each strategy has a cost, such as missed cases in people without a recognised risk factor, a lower sensitivity, or the need for a second test. A common trap is to suggest lowering the cutpoint, which raises Se but lowers Sp and therefore lowers PV+. Worked Example 5.11 applies Equation 5.8 to a proposal for universal HIV screening; notice how low the prevalence is relative to the false positive fraction.
Worked Example 5.11: Universal HIV Screening
A country considers implementing universal HIV screening using a rapid test with Se = 99.5% and Sp = 99.8%. The national HIV prevalence is 0.3%.
PV+ = (0.003 × 0.995) / [(0.003 × 0.995) + (0.997 × 0.002)] = 0.002985 / (0.002985 + 0.001994) = 60.0%
Even with an excellent test (99.5% Se, 99.8% Sp), 40% of positive results in this low-prevalence population would be false positives. This is why confirmatory testing is essential.
Worked Example 5.11 shows why positive results in a low-prevalence population need confirmation, and confirmatory testing is a particular case of combining two tests.
5.1 Combining Two Tests: Series and Parallel
When two tests are used together, the rule for calling a person positive decides what happens to the overall sensitivity and specificity. In series testing, a positive result on the first test triggers the second, and a person is classified as positive only if both tests are positive. In parallel testing, both tests are done at once, and a person is classified as positive if either test is positive. If the errors of the two tests are independent (a person missed by one test is no more likely than anyone else to be missed by the other), the combined values are:
- In series: combined Se = Se1 × Se2, and combined Sp = 1 − (1 − Sp1)(1 − Sp2).
- In parallel: combined Se = 1 − (1 − Se1)(1 − Se2), and combined Sp = Sp1 × Sp2.
The logic is the same in both cases. In series, a diseased person must be caught by both tests to be called positive, so sensitivity falls, while a healthy person is called positive only if both tests make a false positive error, so specificity rises. In parallel, a diseased person is missed only if both tests miss them, so sensitivity rises, while a healthy person needs only one false positive to be called positive, so specificity falls. These directions give a quick check on your arithmetic: in series the combined Se should be lower and the combined Sp higher than either test’s alone, and in parallel the reverse. Worked Example 5.12 applies both rules to the same pair of tests and then uses Equation 5.8 to compare the resulting predictive values.
Worked Example 5.12: Two Tests in Series and in Parallel
A first test has Se1 = 0.85 and Sp1 = 0.95, and a second test has Se2 = 0.80 and Sp2 = 0.98. Assume that their errors are independent.
- Series: Se = 0.85 × 0.80 = 0.68; Sp = 1 − (0.05 × 0.02) = 1 − 0.001 = 0.999.
- Parallel: Se = 1 − (0.15 × 0.20) = 1 − 0.03 = 0.97; Sp = 0.95 × 0.98 = 0.931.
At a prevalence of 2%, Equation 5.8 gives a PV+ of 0.26 for the first test alone, 0.93 for the two tests in series and 0.22 for the two tests in parallel. Series testing removes most of the false positives but misses almost a third of the cases; parallel testing misses only 3% of the cases but produces more false positives than either test alone.
The choice between the two strategies is a choice about which mistake matters more. Series testing suits screening in a low-prevalence population, where false positives are the main problem and a confirmatory test is applied to the first-stage positives. Parallel testing suits a setting where a missed case would be serious, for example a patient whose treatment would change if the disease were present, and the extra false positives can be resolved by further testing. The Final Knowledge Assessment applies the series and parallel formulas to a two-stage newborn hearing screening programme.
This section has shown that the predictive values of a test depend on the prevalence of the condition as well as on its sensitivity and specificity, so that in a low-prevalence population even a highly specific test can produce mostly false positive results, and that combining tests changes the balance of errors. The key takeaways and knowledge check below review this material, and the next section turns to tests whose results lie on a continuous scale.
Key Takeaways
- 4.1 Predictive values answer the question a patient actually asks: given my test result, how likely is it that I do, or do not, have the disease? PV+ is the probability of disease given a positive result, p(D+|T+), and PV− is the probability of no disease given a negative result, p(D−|T−). Sensitivity runs the other way, p(T+|D+): it starts from people known to have the disease and asks how often the test is positive, which is the question from the test’s point of view. The surest way to keep the four measures apart is to name the group in the denominator: everyone with the disease for Se, everyone without it for Sp, everyone who tested positive for PV+ and everyone who tested negative for PV−. When a question asks what a measure means in plain words, start your sentence from that group, for example “of the people who test positive on the norovirus EIA, 95.9% truly have norovirus”.
- 4.2 You can calculate predictive values in three ways, and all three give the same answer. From a 2×2 table, PV+ = a / n1 (true positives divided by everyone who tested positive) and PV− = d / n0 (true negatives divided by everyone who tested negative); in the norovirus study, PV+ = 71/74 = 95.9% and PV− = 103/114 = 90.4%. When you are given only Se, Sp and the prevalence, use the Bayesian formulas (Equations 5.8 and 5.9), in which the numerator is the true positives (or true negatives) and the denominator is everyone with that result, for example PV+ = p(D+)Se / [p(D+)Se + p(D−)(1 − Sp)]. If the formula feels abstract, natural frequencies are a reliable check: take a round number of people such as 10,000, split them by the prevalence, apply Se to the diseased and 1 − Sp to the healthy, and count, as in Worked Example 5.10 (99 true positives and 495 false positives, so PV+ = 99/594 = 16.7%). Whichever method you use, write down the true positive and false positive counts before you divide, because questions often ask for them, and check that they add up to the number who tested positive.
- 4.3 Predictive values depend on the prevalence of disease in the group tested as well as on Se and Sp, which is why the same test can be very useful in one setting and misleading in another. As prevalence falls, PV+ falls and PV− rises: with the norovirus EIA’s Se of 86.6% and Sp of 97.2%, PV+ is 96.9% at a prevalence of 50% but only 3.0% at 0.1% (Table 5.3). The reason is arithmetic, not a fault in the test: when the healthy group is very large, even a small false positive fraction produces more false positives than there are true cases. So when two settings with different prevalence give different PV+ values, explain the difference by prevalence, and do not conclude that the test performs worse in the low-prevalence setting, because its Se and Sp have not changed. For the same reason, a predictive value from one population should not be carried over to another population with a different prevalence.
- 4.4 Because a low PV+ comes from false positives among the many healthy people tested, you raise it by reducing those false positives or by testing a group in which the disease is more common. The lesson describes three strategies: target high-risk groups (which raises the pre-test probability), use a more specific test or a cutpoint that raises Sp (usually at some cost in Se), and confirm positive results with a second test in series. Worked Example 5.11 shows why confirmation matters: even an HIV test with Se = 99.5% and Sp = 99.8% has a PV+ of only 60% at a prevalence of 0.3%. A common trap is to suggest lowering the cutpoint, which raises Se but lowers Sp and therefore lowers PV+. When you recommend a strategy, say how it raises PV+ and what it costs, such as more missed cases or the need for a second test.
- 5.1 When two tests are combined, the rule for calling someone positive decides what happens to sensitivity and specificity. In series, the second test is given only to those positive on the first, and a person is positive only if both tests are positive: combined Se = Se1 × Se2 and combined Sp = 1 − (1 − Sp1)(1 − Sp2), so sensitivity falls while specificity and PV+ rise. In parallel, both tests are done at once and a person is positive if either is positive: combined Se = 1 − (1 − Se1)(1 − Se2) and combined Sp = Sp1 × Sp2, so sensitivity rises and specificity falls. These formulas assume that the two tests make independent errors, and they give a quick check on your arithmetic: in series the combined Se should be lower and the combined Sp higher than either test’s alone, and in parallel the reverse. To choose between them, ask which mistake matters more in the setting: series suits low-prevalence screening, where false positives are the main problem, and parallel suits a setting where a missed case would be serious and the extra positives can be resolved by further testing.
1. A news story about a home test for COVID-19 says: “The test has a sensitivity of 0.85, so a person who tests positive has an 85% chance of being infected.” Which correction is right?
2. A rapid urine test for chlamydia is compared with a laboratory nucleic acid amplification test (NAAT), the reference standard, in 450 clients of a sexual health clinic. Calculate the PV+ and PV− of the rapid test in this group.
| NAAT positive | NAAT negative | Total | |
|---|---|---|---|
| Rapid test positive | 45 | 12 | 57 |
| Rapid test negative | 5 | 388 | 393 |
| Total | 50 | 400 | 450 |
3. The rapid chlamydia test has Se = 0.90 and Sp = 0.97, and in the sexual health clinic, where 11% of the clients tested are infected, its PV+ is 0.79. It is then offered in high schools, where about 2% of the students tested are infected. Use PV+ = (P × Se) ÷ [P × Se + (1 − P)(1 − Sp)] to find the PV+ in the schools, and choose the correct explanation of the change.
4. A health unit screens adults in the general population for hepatitis B with a test that has Se = 0.95 and Sp = 0.98, and finds that most positive results are false. Which change would raise the PV+?
5. A first test has Se = 0.90 and Sp = 0.80, and a second test has Se = 0.80 and Sp = 0.95; assume that their errors are independent. In series (positive only if both tests are positive), Se = Se1 × Se2 and Sp = 1 − (1 − Sp1)(1 − Sp2). In parallel (positive if either test is positive), Se = 1 − (1 − Se1)(1 − Se2) and Sp = Sp1 × Sp2. Which set of combined values is correct?
6. Two programmes each plan to combine two tests. Programme 1 screens donated blood for a virus that can be passed on by transfusion: a missed infected unit could infect a recipient, while a false positive only means that a unit is discarded. Programme 2 screens adults in the community for a rare condition, and a positive result leads to an invasive biopsy. Which pairing of strategies is best?
7. A point-of-care test for antibodies to hepatitis C virus has Se = 0.98 and Sp = 0.99. It is offered at a community health fair, where about 1% of the adults tested have antibodies. For 10,000 adults tested at the fair, which answer gives the correct counts and predictive values, and the correct plain-words meaning of the PV+?
✦ Pass the knowledge check with 100% to continue
Cutpoints, ROC Curves & Likelihood Ratios
⏱ Estimated reading time: 15 minutes
Overview
Sections 1 to 3 judged a test by its accuracy and showed that, in a low-prevalence population, even an accurate test produces many false positives. This section opens by asking when a test should be offered to a whole population as a screening program, a decision that rests on much more than accuracy. It then turns to tests whose results lie on a continuous scale, such as a blood pressure reading, an antibody titre or a probability score, which must be dichotomized at a chosen cutpoint. Every cutpoint trades sensitivity against specificity; the ROC curve and the area under it describe a test across all of its cutpoints, and Youden’s J and the costs of each kind of error guide the choice of one. Finally, likelihood ratios show how much a particular result should change the probability of disease for one person.
Learning Objectives
- 1.2 Apply the Wilson and Jungner criteria, and their revision by Andermann and colleagues, to decide whether a test should become a screening program, and explain how lead-time bias, length-biased sampling and overdiagnosis distort its evaluation.
- 6.1 Explain the trade-off between sensitivity and specificity when choosing a cutpoint.
- 6.2 Describe receiver operating characteristic (ROC) curves and the area under the curve (AUC), and interpret an AUC in plain words.
- 6.3 Calculate Youden’s J at each of several cutpoints, identify the cutpoint it favours, and choose a lower or higher cutpoint when false negatives or false positives are more costly.
- 7.1 Define and calculate likelihood ratios for positive and negative test results and for categories of a result.
- 7.2 Apply likelihood ratios to update pre-test probability to post-test probability.
1.2 From a Test to a Screening Program
Offering a test to a whole population as a screening program raises questions about the condition, its treatment and the program itself, as well as about the test. The principles used to decide whether screening is worthwhile are set out first, then applied to colorectal cancer screening in Canada, and finally three biases that distort the evaluation of screening programs are described.
1.2.1 Principles for Screening
Newborn screening for phenylketonuria (PKU) is a familiar example of screening that works. The recall below summarises it from HSCI 130, and its retrieval question asks which features of PKU made screening worthwhile, which are the features the principles that follow generalise.
Recall: Newborn Screening for PKU
HSCI 130 Lesson 7, Section 4 (Newborn Screening, Precision Medicine, and DTC Testing), presented newborn screening for phenylketonuria (PKU) as public health genetics that works. PKU can be detected from a heel-prick blood spot before any symptoms appear (the Guthrie assay, 1962), and a phenylalanine-restricted diet started early prevents its harm. Every Canadian province now screens newborns for a panel of conditions.
Retrieval question. Which features of PKU made newborn screening for it worthwhile?
PKU is serious, it can be detected before symptoms by a simple and acceptable test on a blood spot, and an effective treatment, the restricted diet, works best when it starts early.
The accuracy of a test is one requirement for screening among several. A screening program offers a test to people without symptoms, so it does harm to some of them (false positives, unnecessary follow-up and the treatment of disease that would never have caused illness) as well as good to others. Wilson and Jungner (1968), writing for the World Health Organization, set out ten principles for deciding whether screening for a condition is worthwhile:
- The condition sought should be an important health problem.
- There should be an accepted treatment for people with recognized disease.
- Facilities for diagnosis and treatment should be available.
- There should be a recognizable latent or early symptomatic stage.
- There should be a suitable test or examination.
- The test should be acceptable to the population.
- The natural history of the condition, including its development from latent to declared disease, should be adequately understood.
- There should be an agreed policy on whom to treat as patients.
- The cost of case-finding, including diagnosis and treatment, should be economically balanced in relation to possible expenditure on medical care as a whole.
- Case-finding should be a continuing process.
The first eight principles concern the condition, the test and the treatment, and the sensitivity, specificity and predictive values of Sections 2 and 3 are the evidence for the fifth. Andermann and colleagues (2008) reviewed the criteria proposed in the following forty years and synthesized them into a revised set that adds requirements for the program itself: it should respond to a recognized need; define its objectives and its target population at the outset; rest on scientific evidence of effectiveness; integrate education, testing, clinical services and program management; include quality assurance that minimizes the risks of screening; ensure informed choice, confidentiality and respect for autonomy; promote equity and access for the whole target population; plan its evaluation from the outset; and show that its overall benefits outweigh its harms.
1.2.2 Applying the Criteria: Colorectal Cancer Screening
In a scenario, each concern points to a named principle. Worked Example 5.13 applies both sets of criteria to colorectal cancer screening in Canada, first through the principles that concern the condition, the test and the treatment and then through the program-level criteria. As you read it, match each sentence to the principle or criterion it illustrates.
Worked Example 5.13: Applying the Screening Criteria to Colorectal Cancer in Canada
Colorectal cancer is one of the most commonly diagnosed cancers in Canada and a leading cause of cancer death, so it is an important health problem. Most colorectal cancers develop over years from adenomatous polyps, which gives a recognizable early stage, and polyps can be removed during colonoscopy, which gives an accepted treatment. The fecal immunochemical test (FIT), which detects blood in a stool sample collected at home, is a suitable and acceptable test, and the Canadian Task Force on Preventive Health Care (2016) recommends screening adults aged 50 to 74 at average risk with a fecal test every two years or with flexible sigmoidoscopy every ten years. Organized provincial programs offer the test to people in the target age range and recall them when the next test is due, which makes case-finding a continuing process.
The program-level criteria show where such programs succeed or struggle. A positive FIT leads to colonoscopy, so the facilities criterion depends on colonoscopy capacity: long waits after a positive result reduce the benefit of early detection. Quality assurance covers the follow-up of every positive result and the standards for colonoscopy, whose rare complications include bleeding and perforation. Participation varies across population groups, so the equity criterion requires that programs measure who is screened and reach those who are not. Randomized trials of fecal occult blood testing have shown reductions in colorectal cancer mortality, which supports the judgement that benefits outweigh harms for the target age range.
Worked Example 5.13 shows that a suitable test is necessary but not sufficient: long waits for colonoscopy after a positive FIT, for instance, point to the facilities criterion even though the test itself performs well.
1.2.3 Biases in Evaluating Screening
Screening programs are judged by outcomes that three biases can distort, and each of them can make screening look more helpful than it really is. Lead-time bias arises because screening moves the date of diagnosis earlier. Survival is usually counted from diagnosis, so screen-detected cases appear to survive longer simply because the clock started sooner, even when death is not delayed at all. Length-biased sampling arises because screening is done at intervals, such as every two years. A fast-growing cancer may pass through its detectable but symptom-free stage between two screens and be found only when symptoms appear, while a slow-growing cancer stays in that stage for years and is very likely to be caught by a screen. Screen-detected cases are therefore weighted toward slow-growing disease, which has a better prognosis anyway, so they look healthier than symptom-detected cases even if screening does nothing. Overdiagnosis is the extreme form of this problem: the detection of disease that would never have caused symptoms or death in the person's lifetime. Each overdiagnosed person is counted as a screen-detected case who "survived", which flatters the program, although that person gained nothing from the diagnosis and may have been harmed by treatment.
A simple timeline shows lead-time bias. Suppose a cancer would cause symptoms at age 67 and death at age 70, so a person diagnosed after symptoms survives 3 years from diagnosis. If screening finds the same cancer at age 64 and treatment does not change its course, the person still dies at 70, but survival from diagnosis is now 6 years: survival has doubled although no death has been delayed.
All three biases distort a comparison of screen-detected cases with symptom-detected cases, so a sound evaluation avoids that comparison. Lead-time bias lengthens survival measured from diagnosis without changing when people die, so the evaluation compares mortality, the number of deaths from the condition in the population, rather than the survival of cases. Length-biased sampling and overdiagnosis fill the screen-detected group with slow-growing or harmless disease, so the evaluation compares whole groups rather than cases: people randomly assigned to be offered screening against people who are not, counting every case and every death in each group, including cancers that appear between screens. In that comparison, overdiagnosed cases add diagnoses to the screened group but add no deaths, and slow-growing cancers occur in both groups, so neither bias can make screening look better than it is. For these reasons a screening program is evaluated by its effect on mortality from the condition in the whole screened population, ideally in a randomized trial.
The fifth Wilson and Jungner principle asks for a suitable test. For many screening tests, such as FIT, whose result is a concentration, suitability depends on where the line between positive and negative is drawn.
6.1 The Cutpoint and the Trade-off Between Sensitivity and Specificity
Many tests produce results on a continuous or semi-quantitative scale, such as blood urea nitrogen levels, optical density values or enzyme activity. To classify individuals as positive or negative, we select a cutpoint (also called a cut-off or threshold) at or above which a result is called positive. The choice of cutpoint matters because the test values of healthy and diseased people rarely separate cleanly.
The Overlap Problem
In reality, the distributions of test values for healthy and diseased individuals often overlap: some healthy people have fairly high values and some diseased people have fairly low ones. Whatever cutpoint we choose will therefore result in both false positive and false negative results. Raising the cutpoint makes the test harder to pass as positive. Fewer healthy people reach it, so Sp rises (fewer false positives), but some diseased people with modest values now fall below it, so Se falls (more false negatives). Lowering the cutpoint has the opposite effect, catching more of the diseased at the cost of calling more healthy people positive. No cutpoint improves both at once; it can only move errors from one kind to the other.
Figure 5.4 shows the overlap and a cutpoint that divides it. Healthy people whose values fall above the cutpoint are false positives, and diseased people whose values fall below it are false negatives, so a single line cannot separate two groups whose values share a range.
Figure 5.4. Overlap between healthy and diseased distributions. Moving the cutpoint left or right trades off sensitivity for specificity.
Each cutpoint therefore gives its own pair of sensitivity and specificity values, and a quick check on any table of cutpoints is that sensitivity should fall and specificity rise as the cutpoint goes up. A single graph can display all of these pairs together.
6.2 ROC Curves and the Area Under the Curve
A receiver operating characteristic (ROC) curve plots the Se (y-axis) against the false positive fraction, 1 − Sp (x-axis), computed at a number of different cutpoints (Hanley & McNeil, 1982; see also Wikipedia: ROC curve). Each cutpoint gives one pair of values, so each cutpoint becomes one point on the graph, and joining the points traces the curve. Because the whole range of cutpoints is shown at once, the curve is used both to evaluate the overall performance of a test, without committing to any one cutpoint, and to select a cutpoint. The best possible point is the top-left corner, where sensitivity is 100% and the false positive fraction is 0, so a curve that bows towards the top-left corner belongs to a test that discriminates well. Expand the two panels below to see how to read the curve against its reference lines and how parametric and non-parametric curves are drawn.
The 45° diagonal line represents a test with no discriminating ability (no better than chance). The closer the ROC curve gets to the top-left corner, the better the test discriminates between D+ and D− individuals. The top-left corner represents a test with Se = 100% and Sp = 100%.
There are two ways to draw the curve. A non-parametric ROC curve makes no assumptions: it simply uses each observed test value as a cutpoint in turn, plots the Se and (1 − Sp) it produces, and joins the points, which gives a curve made of small steps. A parametric ROC curve instead assumes a shape for the test values in each group, usually that the values of healthy people and the values of diseased people each follow a normal (bell-shaped) distribution, which is called a binormal model. From those two assumed distributions it calculates a smooth curve. The smooth curve is easier to read, but it is only as good as the assumption about the shape of the data. Both approaches can generate 95% confidence intervals.
Each cutpoint is one point on the curve. A very low cutpoint calls almost everyone positive, so both Se and 1 − Sp are close to 1 and its point lies near the top-right corner. Raising the cutpoint lowers both, so a higher (stricter) cutpoint sits further down and to the left on the curve, towards the bottom-left corner, which is reached when the cutpoint is above every test value and no one is called positive.
The area under the curve (AUC) summarises the overall discriminatory ability of the test across all cutpoints. It can be interpreted as the probability that a randomly selected D+ individual has a greater test value than a randomly selected D− individual, which is equivalent to the Mann–Whitney U statistic (Hanley & McNeil, 1982). This interpretation explains the two reference values. A test that carries no information about disease would give the diseased person the higher value only half the time, as a coin toss would, so its AUC is 0.5 and its curve lies along the diagonal. A test whose values for diseased people are all higher than those for healthy people would rank every pair correctly, so its AUC is 1, which is perfect separation. Most real tests fall somewhere in between. Table 5.4 gives conventional labels for ranges of the AUC.
Table 5.4. Conventional interpretation of the area under the ROC curve (AUC).
| AUC Value | Interpretation |
|---|---|
| 0.50 | No discrimination (chance alone) |
| 0.50 – 0.70 | Poor discrimination |
| 0.70 – 0.80 | Acceptable discrimination |
| 0.80 – 0.90 | Excellent discrimination |
| > 0.90 | Outstanding discrimination |
Interpreting an AUC
State the probability and then the label from Table 5.4: an AUC of 0.85 means that if one diseased and one healthy person are chosen at random, there is an 85% chance that the diseased person has the higher value, which is excellent discrimination. The AUC is not the proportion of people classified correctly, which depends on the cutpoint and on the prevalence, and it does not change when the cutpoint moves.
Interactive 5.2 builds a ROC curve from two overlapping score distributions and reports the sensitivity, the false positive fraction, Youden’s J index (introduced in 6.3) and the AUC. Drag the cutoff and notice that the point moves along the curve while the AUC stays the same; then increase the separation of the distributions or reduce their spread and watch the curve bow towards the upper-left corner as the AUC rises.
📊 Interactive 5.2: ROC Curve Builder
Test score distributions
Drag the dashed cutoff line.
ROC curve
Yellow dot = current cutoff. Diagonal = random-chance reference.
The ROC curve shows every cutpoint at once, but a test in use needs one cutpoint.
6.3 Choosing a Cutpoint with Youden’s J
If sensitivity and specificity are given equal weight, the optimal cutpoint is where Se + Sp is at a maximum, which corresponds to the point on the ROC curve farthest from the 45° line. This maximised value of Se + Sp − 1 is known as Youden’s J index. Subtracting 1 gives J a convenient scale. A test that is no better than chance has Se + Sp = 1 at every cutpoint (calling everyone positive, for example, gives Se = 1 and Sp = 0), so its J is 0, while a perfect test has Se + Sp = 2 and a J of 1. J can therefore be read as how far the test at that cutpoint rises above chance, and it is the quantity Interactive 5.2 reports as you drag the cutoff. Youden’s J can be calculated from any table that gives the sensitivity and specificity at several cutpoints: calculate J at each cutpoint and choose the one where it is largest. When a test reports results in ordered categories, the sensitivity and specificity at a cutpoint are found by counting every category at or above the cutpoint as positive and every category below it as negative, which turns the table into a 2×2 table at that cutpoint. Worked Example 5.14 applies these steps to a blood marker evaluated at four cutpoints; notice how close the two best values of J are, and how the costs of errors then decide the choice.
Worked Example 5.14: Choosing a Cutpoint with Youden’s J
A blood marker is evaluated at four cutpoints, with a value at or above the cutpoint called positive.
| Cutpoint (value ≥) | 20 | 30 | 40 | 50 |
|---|---|---|---|---|
| Sensitivity | 0.98 | 0.93 | 0.81 | 0.62 |
| Specificity | 0.60 | 0.80 | 0.91 | 0.97 |
| Youden’s J = Se + Sp − 1 | 0.58 | 0.73 | 0.72 | 0.59 |
J is largest (0.73) at a cutpoint of 30, so that is the cutpoint J favours when sensitivity and specificity are given equal weight. Reading the table from left to right shows the trade-off: each higher cutpoint gives a higher specificity and a lower sensitivity. If a missed case were much more costly than a false alarm, for example a serious infection that is easily treated, a lower cutpoint such as 20 would be chosen, accepting more false positives to raise the sensitivity to 0.98. If a false positive led to a risky or expensive follow-up, a higher cutpoint such as 40 would be chosen instead.
Worked Example 5.14 ends by moving away from the Youden point, because giving sensitivity and specificity equal weight is not as neutral as it sounds. The reason is that sensitivity and specificity apply to groups of different sizes. Sensitivity describes errors among the diseased, and specificity describes errors among the healthy. When a disease is uncommon, a one-point gain in specificity prevents far more errors, in absolute numbers, than a one-point gain in sensitivity, simply because there are many more healthy people. Weighting the two percentages equally therefore weights the two kinds of mistake unequally. Put more formally, the Youden point minimises the expected cost of errors only when the cost of a false negative multiplied by the prevalence equals the cost of a false positive multiplied by (1 − prevalence). That means equal costs for each kind of error only when the prevalence is 50%, so that the two groups are the same size. At a prevalence of 10%, the condition reads cost(FN) × 0.10 = cost(FP) × 0.90, so the Youden point quietly treats one missed case as costing as much as nine false positives. J is therefore a starting point rather than the final answer: choose a lower cutpoint to raise sensitivity when a missed case costs more than that, and a higher cutpoint to raise specificity when a false positive leads to a risky or expensive follow-up, stating what the move gains and what it costs. Expand the panel below for a summary of this cost condition and for a second criterion, the point closest to the top-left corner.
The point closest to the top-left corner is a separate criterion from Youden’s J; it agrees with the Youden point when the ROC curve is symmetric and can differ from it when the curve is lopsided. Equal weight on Se and Sp corresponds to equal costs for each false negative and each false positive only when prevalence is 50%. In general, the Youden point minimises the expected cost of errors when the cost of a false negative multiplied by the prevalence equals the cost of a false positive multiplied by (1 − prevalence); at 10% prevalence, for example, it treats one missed case as costing as much as nine false positives. However, if the costs depart from this balance, you might emphasise Se or Sp depending on the clinical context.
The ROC curve, the AUC and Youden’s J describe a test across its cutpoints. A clinician who receives a particular result also needs to know how much that result changes the probability of disease.
7.1 Likelihood Ratios
A likelihood ratio (LR) is the ratio of the probability of a given test result among D+ individuals to the probability of that same result among D− individuals (Deeks & Altman, 2004). Put simply, it asks how much more likely a particular result is in someone with the disease than in someone without it. If a result is common among the diseased and rare among the healthy, seeing it should make us more confident that the person has the disease, and the LR is well above 1. If the result is about equally common in both groups, it tells us little, and the LR is close to 1. If the result is more common among the healthy, it points away from disease, and the LR is below 1. Like sensitivity and specificity, LRs are calculated within the disease groups, so they do not depend on prevalence. They combine the information in Se and Sp into one number, and they convert pre-test odds of disease into post-test odds through Bayes's theorem. There are three kinds: the likelihood ratio for a positive result (Equation 5.10), for a negative result (Equation 5.11) and for a particular category of result (Equation 5.12). Open each tab below to work through one of them. Worked Examples 5.15 and 5.16 apply the first two to the norovirus EIA and Calculators 5.9 and 5.10 reproduce them; Worked Example 5.17 applies the third to a serum marker grouped into three categories, and Calculator 5.11 reproduces it. In the category-specific tab, check that each proportion is divided by the total number of diseased, or of non-diseased, people, not by the total for the category.
Likelihood Ratio for a Positive Test (LR+)
The LR+ compares the chance of a positive result in the diseased (the sensitivity) with the chance of a positive result in the healthy (the false positive fraction). It is also the factor by which a positive result multiplies the odds of disease: the odds of disease after a positive test result, divided by the odds before the test, equal the LR+. Higher LR+ values therefore mean that a positive test result is more informative for confirming disease.
Worked Example 5.15: LR+
For the norovirus EIA, Se = 71/82 = 0.866 and Sp = 103/106 = 0.9717, so 1 − Sp = 0.0283.
LR+ = 0.866 / 0.0283 = 30.6. A positive EIA result is about 31 times as likely in a person with norovirus as in a person without it.
Likelihood Ratio for a Negative Test (LR−)
Lower LR− values mean a negative test result is more informative for ruling out disease. An LR− close to 0 is ideal.
Worked Example 5.16: LR−
For the same test, LR− = (1 − 0.866) / 0.9717 = 0.134 / 0.9717 = 0.138. A negative EIA result is about one-seventh as likely in a person with norovirus as in a person without it, so a negative result divides the odds of infection by roughly seven.
Category-Specific LR
Instead of simply classifying results as positive or negative, researchers in diagnostic settings often calculate category-specific LRs based on the actual test value. This uses the actual result rather than just positive/negative, so the strength of evidence is graded by how extreme the value is.
Worked Example 5.17: Category-Specific LRs
A serum marker is measured in 100 people with a disease and 200 people without it, and the results are grouped into three categories.
| Marker level | D+ (n = 100) | D− (n = 200) | LRcat |
|---|---|---|---|
| High | 60 | 4 | (60/100) / (4/200) = 0.60 / 0.02 = 30.0 |
| Intermediate | 30 | 36 | (30/100) / (36/200) = 0.30 / 0.18 = 1.67 |
| Low | 10 | 160 | (10/100) / (160/200) = 0.10 / 0.80 = 0.125 |
A high result is 30 times as likely in a person with the disease as in a person without it and strongly supports the diagnosis. An intermediate result changes the odds very little, and a low result reduces them to one-eighth. Collapsing the three categories into a single positive or negative result would discard this gradation.
To interpret any likelihood ratio, use the wording of Worked Example 5.15: a positive EIA result is about 31 times as likely in a person with norovirus as in a person without it. The glossary gives rough guides for judging strength: an LR+ above 10 strongly rules disease in, an LR− below 0.1 strongly rules it out, and values near 1 change the odds very little (Deeks & Altman, 2004). The EIA’s LR+ of 30.6 is well above 10, but its LR− of 0.138 is just above 0.1, so it is better at ruling infection in than out. Worked Example 5.17 adds a further point: the high and low marker categories carry strong evidence that collapsing the results into positive and negative would discard.
A likelihood ratio on its own says how strongly a result points towards or away from disease. To turn it into the probability of disease for a particular person, it has to be combined with what was known before testing.
7.2 From Pre-Test to Post-Test Probability
Likelihood ratios update the assessment of disease probability after a test result in three steps, which Equation 5.13 writes as formulae. First convert the pre-test probability, often the prevalence or a clinical estimate, to pre-test odds. Then multiply the odds by the likelihood ratio for the result obtained. Finally convert the post-test odds back to a probability.
Convert to Odds First
A likelihood ratio multiplies odds, not probabilities. Multiplying a probability directly by the LR gives a wrong answer, which can even exceed 1, so always convert the probability to odds before multiplying and convert the odds back afterwards.
Worked Example 5.18 applies the three steps of Equation 5.13 to a pre-test probability of 2% and a result whose category-specific LR is 25.95. Follow the three conversions in order.
Worked Example 5.18: Three-Step Process
- Convert pre-test probability to pre-test odds: odds = P / (1 − P)
- Multiply by the likelihood ratio: post-test odds = pre-test odds × LR
- Convert post-test odds back to probability: P = odds / (1 + odds)
Example: Pre-test probability = 2%, test result at a cutpoint where LRcat = 25.95.
- Pre-test odds = 0.02/0.98 = 0.0204
- Post-test odds = 0.0204 × 25.95 = 0.5294
- Post-test probability = 0.5294 / (1 + 0.5294) = 35%
After obtaining the test result, the estimated probability of disease rises from 2% to 35%.
Worked Example 5.18 shows that a strongly positive result can raise a low pre-test probability a long way and still leave the patient more likely than not to be free of disease. Calculator 5.12 applies the three steps to any pre-test probability and likelihood ratio; try a pre-test probability of 30% with the EIA’s LR+ and LR−.
The same three steps apply to a negative result, using LR−. Suppose a patient’s pre-test probability of norovirus is 30%, so the pre-test odds are 0.30 ÷ 0.70 = 0.429. After a positive EIA (LR+ = 30.6, Worked Example 5.15), the post-test odds are 0.429 × 30.6 = 13.1 and the post-test probability is 13.1 ÷ 14.1 = 93%. After a negative EIA (LR− = 0.138, Worked Example 5.16), the post-test odds are 0.429 × 0.138 = 0.059 and the post-test probability is 0.059 ÷ 1.059 = 5.6%. A positive result therefore rules infection in convincingly, while whether a residual probability of about 6% after a negative result is low enough to stop investigating depends on what a missed case would cost.
The three steps also connect likelihood ratios to the predictive values of Section 3. When the pre-test probability is the prevalence of the study sample, the post-test probability after a positive result is the PV+. In the norovirus study the prevalence was 0.436, so the pre-test odds are 0.436 ÷ 0.564 = 0.773; multiplying by LR+ = 30.6 gives post-test odds of 23.7, and 23.7 ÷ 24.7 = 0.96, the PV+ of 95.9% found in 4.2.1. In the same way, the post-test probability after a negative result equals 1 − PV−: the odds become 0.773 × 0.138 = 0.107, a probability of 0.107 ÷ 1.107 = 0.096, which is 1 − 0.904. Both routes apply Bayes’s theorem, but the likelihood ratio route keeps the test (the LR) separate from the setting (the pre-test odds), so it can be applied to any patient’s own pre-test probability rather than only to the prevalence of the study sample.
This section has set the accuracy of a test among the wider criteria for a screening program, shown how the choice of cutpoint determines sensitivity and specificity, how the ROC curve and the AUC summarise a test across all cutpoints, and how likelihood ratios convert a pre-test probability into a post-test probability. The key takeaways and knowledge check below review this material.
Key Takeaways
- 1.2 An accurate test is only one requirement for a good screening programme, because a programme offers the test to people without symptoms and so harms some of them (false positives, unnecessary follow-up and the treatment of disease that would never have caused illness) while it helps others. The Wilson and Jungner principles ask about the condition (an important health problem with a recognizable early stage and a well-understood natural history), the test (suitable and acceptable to the population), the treatment and the facilities for it, an agreed policy on whom to treat, the cost, and whether case-finding is continuing; Andermann and colleagues added criteria for the programme itself, such as clear objectives, quality assurance, informed choice, equity, a planned evaluation and benefits that outweigh harms. In a scenario, match each concern to the principle it illustrates: long waits for colonoscopy after a positive FIT, for example, point to the facilities criterion (Worked Example 5.13). Be ready to explain the three biases that make screening look better than it is: lead-time bias (earlier diagnosis lengthens survival measured from diagnosis even when death is not delayed), length-biased sampling (screening at intervals tends to find slow-growing disease with a better prognosis) and overdiagnosis (finding disease that would never have caused symptoms or death). Because of these biases, a programme is judged by mortality from the condition in the whole screened population, ideally in a randomized trial, not by the survival of screen-detected cases.
- 6.1 Many tests give a result on a continuous scale, so someone has to choose a cutpoint at or above which the result is called positive. Because the test values of healthy and diseased people overlap (Figure 5.4), every cutpoint leaves some false positives and some false negatives. Raising the cutpoint calls more healthy people negative, so specificity rises and false positives fall, but it also calls more diseased people with low values negative, so sensitivity falls and false negatives rise; lowering the cutpoint does the opposite. When a question asks why no cutpoint removes both kinds of error, the answer is the overlap: a single line cannot separate two groups whose values share a range. A quick check when you read a table of cutpoints is that sensitivity should fall and specificity should rise as the cutpoint goes up.
- 6.2 A receiver operating characteristic (ROC) curve shows the whole trade-off at once by plotting sensitivity (y-axis) against the false positive fraction, 1 − Sp (x-axis), at every cutpoint. The 45° diagonal represents a test no better than chance, and the closer the curve bows towards the top-left corner (Se = 100%, Sp = 100%), the better the test discriminates between diseased and healthy people. The area under the curve (AUC) summarises this across all cutpoints: it is the probability that a randomly chosen person with the disease has a higher test value than a randomly chosen person without it. To interpret an AUC in plain words, use that sentence and then add the label from Table 5.4: an AUC of 0.85, for example, means that if one diseased and one healthy person are chosen at random, there is an 85% chance that the diseased person has the higher value, which is excellent discrimination. Remember that the AUC describes the test as a whole and does not change when you move the cutpoint; moving the cutpoint only moves you to a different point on the same curve.
- 6.3 Youden’s J, J = Se + Sp − 1, is one way to pick a single cutpoint from a table: calculate J at each cutpoint and choose the one where it is largest. In Worked Example 5.14, J is 0.58, 0.73, 0.72 and 0.59 at cutpoints of 20, 30, 40 and 50, so J favours 30. J gives sensitivity and specificity equal weight, so it is a starting point rather than the final answer when the costs of the two kinds of error are clearly unequal. When a missed case is more costly (for example a serious infection that is easily treated), choose a lower cutpoint to raise sensitivity and accept more false positives; when a false positive leads to a risky or expensive follow-up, choose a higher cutpoint to raise specificity. In your answer, name the cutpoint J favours, say which way you would move and why, and quote the change in Se and Sp that the move gains and costs.
- 7.1 A likelihood ratio (LR) compares how likely a particular test result is in people with the disease with how likely it is in people without it. For a positive result, LR+ = Se / (1 − Sp), and for a negative result, LR− = (1 − Se) / Sp; for the norovirus EIA, LR+ = 0.866 / 0.0283 = 30.6 and LR− = 0.134 / 0.9717 = 0.138. A category-specific LR divides the proportion of diseased people with that result by the proportion of healthy people with it (in Worked Example 5.17, a high marker level gives (60/100) / (4/200) = 30.0), which keeps the information that collapsing results into positive and negative would throw away. To interpret an LR, use the wording of Worked Example 5.15, “a positive EIA result is about 31 times as likely in a person with norovirus as in a person without it”, and judge its strength with the rough guides: an LR+ above 10 strongly rules disease in, an LR− below 0.1 strongly rules it out, and values near 1 change the odds very little. A common slip with category-specific LRs is to divide by the total for that category; each proportion should instead use the total number of diseased, or of healthy, people as its denominator.
- 7.2 Likelihood ratios let you turn what was believed before testing (the pre-test probability, often the prevalence or a clinical estimate) into a post-test probability for one person, in three steps. First convert the probability to odds, odds = P / (1 − P); then multiply the odds by the LR for the result obtained; then convert back, P = odds / (1 + odds). For a pre-test probability of 30% and the norovirus EIA, the pre-test odds are 0.429, so a positive result gives odds of 13.1 and a probability of 93%, and a negative result gives odds of 0.059 and a probability of 5.6%. The step most often skipped is the conversion: multiplying a probability directly by the LR gives a wrong answer, and can even give a “probability” above 1. When you finish, say whether the result rules disease in or out and whether the remaining probability is low enough to act on, which depends on what a missed case would cost.
1. A clinic raises the fasting glucose cutpoint for calling a result positive for diabetes from 6.1 to 7.0 mmol/L. The fasting glucose values of people with and without diabetes overlap. What happens?
2. Two blood markers are evaluated for the same infection. The ROC curve of marker X lies close to the 45° diagonal (AUC = 0.54). The curve of marker Y bows towards the top-left corner (AUC = 0.88). Which statement is correct?
3. A skin meter for newborn jaundice is evaluated against a blood bilirubin test at four cutpoints, with a reading at or above the cutpoint called positive. Missing a baby with severe jaundice can cause permanent brain damage, whereas a false positive leads only to a heel-prick blood test. Which answer is correct?
| Cutpoint (reading ≥, µmol/L) | 150 | 200 | 250 | 300 |
|---|---|---|---|---|
| Sensitivity | 0.98 | 0.93 | 0.84 | 0.66 |
| Specificity | 0.60 | 0.78 | 0.91 | 0.97 |
4. A test has Se = 0.80 and Sp = 0.95. Using LR+ = Se ÷ (1 − Sp) and LR− = (1 − Se) ÷ Sp, which answer is correct?
5. For the test in the previous question (LR+ = 16, LR− = 0.21), a physician judges a patient’s probability of disease to be 20% before testing. What is the probability of disease after a positive result and after a negative result?
6. A regional programme screens adults for a slow-growing cancer. Its annual report says: “Since screening began, twice as many cancers are diagnosed each year, and people whose cancer was found by screening live years longer after diagnosis than people diagnosed after symptoms, which proves that screening works.” Death rates from the cancer in the region have not changed. Which response is best?
7. A family practice screens adults for major depression with a questionnaire, and a positive screen leads to a clinical interview. At the chosen cutpoint, Se = 0.88 and Sp = 0.85, so LR+ = 0.88 ÷ 0.15 = 5.87. About 10% of the practice’s adult patients have major depression. What is the probability of depression for a patient who screens positive?
✦ Pass the knowledge check with 100% to continue
Final Review & Assessment
⏱ Estimated time: 20 minutes
Bringing It All Together
This lesson built up the toolkit for evaluating tests, from the basic distinction between screening (in healthy populations) and diagnosis (in suspected cases), through sensitivity and specificity, into predictive values, and finally into the more sophisticated machinery of cutpoints, ROC curves, and likelihood ratios. The arc moves from how does the test perform? to what does this result mean for this patient in this setting?
The deepest idea in the lesson is that test performance is never just a property of the test. The same sensitivity and specificity produce very different predictive values when prevalence changes, which is why a screening protocol that works in a high-prevalence clinic can collapse into mostly false positives when applied to the general population. Published diagnostic-accuracy studies themselves are subject to design-related bias that inflates reported performance (Lijmer et al., 1999), motivating the QUADAS-2 quality-assessment tool (Whiting et al., 2011) and STARD 2015 reporting standard (Bossuyt et al., 2015). As you finish the assessment, the takeaways below are the practical companions: keep them in mind whenever someone tells you a test is “accurate.”
Key Takeaways from this lesson
- You can now tell a screening use of a test, in apparently healthy people whose positive results are sent on for work-up, from a diagnostic use in people already suspected of disease, and you can judge a proposed screening programme against the Wilson and Jungner principles and the criteria of Andermann and colleagues. You can also explain how lead-time bias, length-biased sampling and overdiagnosis make screen-detected cases look as if they do better than they really do. Remember that a programme is judged by mortality from the condition in the whole screened population, not by survival after diagnosis, and that each concern in a scenario (waiting lists, acceptability, cost) points to a named principle.
- You can now separate analytic sensitivity, a laboratory detection limit, from diagnostic sensitivity, describe a test’s accuracy (how close the average is to the truth) and precision (how consistent the results are), and calculate and interpret a coefficient of variation, limits of agreement and Cohen’s kappa. For kappa, work step by step (po, then pe, then κ), give its label from Table 5.1, and remember that prevalence and rater bias can pull it down. Above all, keep in mind that close agreement and a small CV show precision only: a test can be consistent and still be consistently wrong.
- You can now set out a 2×2 table against a reference standard, calculate sensitivity, specificity and the false negative and false positive fractions, read a confidence interval for them, and explain why they are properties of the test (SnNOut for ruling out, SpPIn for ruling in) that can still shift with the spectrum of disease. You can also explain why the proportion testing positive, the apparent prevalence, differs from the true prevalence, and correct it with the Rogan-Gladen formula. The most useful reminder for the exam is to divide by the disease-status totals (everyone with the disease for Se, everyone without it for Sp), never by the number who tested positive or negative.
- You can now calculate PV+ and PV− from a 2×2 table, from the Bayesian formulas and by counting natural frequencies, and explain why they change with prevalence while sensitivity and specificity do not. You can also name the strategies that raise PV+: testing higher-risk groups, using a more specific test and confirming positive results with a second test in series. When a question gives a low prevalence, count out 10,000 people and compare the false positives with the true positives, because that comparison explains a low PV+ more clearly than any formula.
- You can now calculate the combined sensitivity and specificity of two tests used in series (positive only if both are positive) and in parallel (positive if either is positive), and choose the strategy that suits the setting. Remember the direction of each change: series lowers Se and raises Sp and PV+, while parallel raises Se and lowers Sp. Then justify your choice by the mistake that matters more, false positives in low-prevalence screening or missed cases where a missed diagnosis would be serious.
- You can now explain why moving a cutpoint trades sensitivity against specificity, calculate Youden’s J at several cutpoints and find the one it favours, read a ROC curve, and interpret its AUC as the probability that a randomly chosen diseased person has a higher test value than a randomly chosen healthy person. Remember that J weights Se and Sp equally, so when a missed case is more costly you move to a lower cutpoint, and when a false positive is more costly you move to a higher one.
- You can now calculate LR+, LR− and category-specific likelihood ratios, interpret each in a sentence, and use them to move from a pre-test probability to a post-test probability for one person. The most important reminder is to convert the probability to odds before multiplying by the LR and to convert the odds back to a probability afterwards, and then to say whether the result rules disease in or out (an LR+ above 10, or an LR− below 0.1, is strong evidence).
Final Knowledge Assessment
Complete all 17 questions below with 100% accuracy to finish this lesson. The final assessment can be submitted once every section knowledge check has been passed.
1. A province offers a point-of-care syphilis test to every pregnant person at the first prenatal visit, and anyone who tests positive then has laboratory serology. At a sexual health clinic, a physician orders laboratory serology for a man with a painless genital ulcer. How should each use of testing be classified?
2. A control serum is tested 10 times on a laboratory analyser, giving a mean of 4.00 mmol/L and a standard deviation of 0.12 mmol/L. A technologist says: “The coefficient of variation is small, so the analyser gives the right answer.” Which response is correct?
3. In clinics without a reading device, the point-of-care syphilis strip is read by eye. Two technologists independently read the same 150 strips, with the results below. Calculate Cohen’s kappa and choose the correct conclusion.
| Technologist B: positive | Technologist B: negative | Total | |
|---|---|---|---|
| Technologist A: positive | 24 | 6 | 30 |
| Technologist A: negative | 6 | 114 | 120 |
| Total | 30 | 120 | 150 |
4. Two physiotherapists independently rate the gait of the same 100 older adults as normal, mildly impaired or severely impaired. Cohen’s (unweighted) kappa is 0.58, and with linear weights the weighted kappa is 0.69. Most of their disagreements are between neighbouring categories. Which explanation is correct?
5. The point-of-care syphilis test is evaluated against laboratory serology (the reference standard) in 500 clients of a sexual health clinic. The columns give the reference result and the rows the point-of-care result. Which set of values is correct?
| Syphilis (laboratory) | No syphilis (laboratory) | Total | |
|---|---|---|---|
| Point-of-care positive | 88 | 16 | 104 |
| Point-of-care negative | 12 | 384 | 396 |
| Total | 100 | 400 | 500 |
6. In the clinic study (prevalence of syphilis 20%), the point-of-care test had Se = 0.88, Sp = 0.96 and PV+ = 0.85. A colleague proposes to quote all three values as properties of the test in guidance for prenatal clinics, where far fewer of the people tested have syphilis. Which response is correct?
7. In prenatal clinics, where about 0.5% of pregnant people have syphilis, the point-of-care test (Se = 0.88, Sp = 0.96) is positive in 4.4% of those tested. A provincial dashboard reports the 4.4% as “the prevalence of syphilis in pregnancy”. Which statement is correct?
8. For 10,000 pregnant people screened with the point-of-care syphilis test (Se = 0.88, Sp = 0.96) at a prevalence of 0.5%, what are the expected numbers of true positives and false positives, and the PV+?
9. Laboratory serology has Se = 0.95 and Sp = 0.99 for syphilis, and the point-of-care test has Se = 0.88 and Sp = 0.96; assume that their errors are independent. In series, Se = Se1 × Se2 and Sp = 1 − (1 − Sp1)(1 − Sp2); in parallel, Se = 1 − (1 − Se1)(1 − Se2) and Sp = Sp1 × Sp2. Which statement is correct?
10. A blood marker for liver fibrosis is evaluated at four cutpoints, with a score at or above the cutpoint called positive. A positive result leads to a liver biopsy, which carries a risk of serious bleeding, while the disease progresses slowly and a missed case can be found at the next yearly test. Which answer is correct?
| Cutpoint (score ≥) | 1.0 | 1.5 | 2.0 | 2.5 |
|---|---|---|---|---|
| Sensitivity | 0.96 | 0.89 | 0.78 | 0.60 |
| Specificity | 0.55 | 0.72 | 0.86 | 0.95 |
11. A study reports that a new urine test for bladder cancer has an area under the ROC curve (AUC) of 0.76. A colleague writes: “The test gives the correct result for 76% of patients.” Which interpretation is correct?
12. A man with a genital ulcer and a recent partner with syphilis has a pre-test probability of syphilis of 40%. The point-of-care test has Se = 0.88 and Sp = 0.96, so LR+ = 0.88 ÷ 0.04 = 22 and LR− = 0.12 ÷ 0.96 = 0.125. What is his probability of syphilis after a positive result and after a negative result?
13. For the same man (LR+ = 22, LR− = 0.125, probability of syphilis 7.7% after a negative result), is the point-of-care test better at ruling syphilis in or ruling it out, and should a negative result rule it out? As a rough guide, an LR+ above 10 strongly rules disease in and an LR− below 0.1 strongly rules it out.
14. C-reactive protein (CRP) is measured in 400 young children with fever, 80 of whom have a serious bacterial infection on the reference work-up. A child with fever whose pre-test probability of serious infection is 10% has a high CRP. What is the likelihood ratio for a high result, and the child’s post-test probability?
| CRP result | Serious infection (n = 80) | No serious infection (n = 320) |
|---|---|---|
| High (80 mg/L or more) | 48 | 16 |
| Intermediate (20 to 79 mg/L) | 24 | 96 |
| Low (below 20 mg/L) | 8 | 208 |
15. A ministry proposes screening adults aged 40 to 70 for thyroid cancer with a neck ultrasound, followed by a needle biopsy for anyone with a suspicious nodule. Planners learn that the waiting list for biopsies is already eight months long, and that specialists disagree about whether small, slow-growing cancers found this way should be treated at all. Which Wilson and Jungner principles do these two concerns illustrate?
16. A province screens every newborn for permanent hearing loss, which affects about 2 in 1,000 newborns. For 10,000 newborns, the first test alone (otoacoustic emissions) gives about 19 true positives and 998 false positives. Retesting the babies who fail it with a second test, and referring only those who fail both (series testing), gives about 18 true positives and 30 false positives. Which statement is correct?
17. For the same newborn hearing programme, a colleague proposes giving every baby both tests and referring any baby who fails either one (parallel testing), so that fewer babies with hearing loss are missed. For 10,000 newborns this gives about 20 true positives and 1,267 false positives, against about 18 true positives and 30 false positives in series. Which assessment is correct?
✦ Pass every section knowledge check before submitting