HSCI 341 · Lesson 5

Screening &
Diagnostic Tests

Fundamental Epidemiological Concepts and Approaches

Learning objectives for this lesson:

  1. Distinguish screening from diagnostic uses of a test, and judge whether a test should become a screening programme using the Wilson and Jungner principles and the screening biases (lead-time bias, length-biased sampling and overdiagnosis).
  2. Distinguish analytic from diagnostic sensitivity, describe a test’s accuracy and precision, and calculate and interpret measures of precision and agreement (the coefficient of variation, the limits of agreement and Cohen’s kappa).
  3. Calculate sensitivity, specificity and the false negative and false positive fractions from a 2×2 table, interpret them as properties of the test (including a reported confidence interval and the spectrum effect), and explain and estimate the difference between true and apparent prevalence.
  4. Calculate predictive values from a 2×2 table, the Bayesian formulas and natural frequencies, explain why they depend on prevalence when sensitivity and specificity do not, and identify strategies that raise the positive predictive value.
  5. Calculate the combined sensitivity and specificity of two tests used in series or in parallel, and choose the strategy that suits the setting.
  6. Explain the trade-off between sensitivity and specificity as a cutpoint moves, calculate Youden’s J, interpret a ROC curve and its area under the curve (AUC), and choose a cutpoint for a given setting.
  7. Calculate and interpret likelihood ratios for positive, negative and category-specific results, and use them to update a pre-test probability to a post-test probability.

This course was developed by Dr. Kiffer G. Card based on Dohoo, I. R., Martin, S. W., & Stryhn, H. (2012). Methods in Epidemiologic Research. VER Inc.

Reference

Glossary: Key Terms, People & Concepts

📚 Reference page, available throughout the lesson

This glossary collects the key concepts, people, and ideas you will meet in this lesson. Use it as a reference while you work through the material, or as a review before assessments. Type in the search box to filter entries.

Test Performance Concepts
Screening Test A test applied to asymptomatic individuals to identify those at higher risk of disease so they can undergo further diagnostic evaluation. Goal is early detection in apparently healthy people.
Diagnostic Test A test used in symptomatic individuals or those with positive screens to confirm or rule out disease. Generally more invasive, expensive, and accurate than screening tests.
Gold (Reference) Standard The best available test or set of criteria used to define true disease status when evaluating a new test. The benchmark against which sensitivity and specificity are measured.
Accuracy How close a measurement is to the true value. In test evaluation, the proportion of all results (positive and negative) that are correct.
Precision The reproducibility or repeatability of measurements: how close repeated measurements are to one another, regardless of accuracy.
Sensitivity (True Positive Rate) The probability that a test correctly identifies a person with the disease: P(test+ | disease+). High sensitivity is needed to rule out disease (SnNout).
Specificity (True Negative Rate) The probability that a test correctly identifies a person without the disease: P(test− | disease−). High specificity is needed to rule in disease (SpPin).
Positive Predictive Value (PPV) Among those who test positive, the proportion who actually have the disease: P(disease+ | test+). Strongly dependent on the prevalence in the tested population.
Negative Predictive Value (NPV) Among those who test negative, the proportion who truly do not have the disease: P(disease− | test−).
Prevalence Threshold The point along the prevalence axis below which the predictive value of a test deteriorates rapidly. A reminder that a test’s usefulness depends on the population it is applied to.
False Positive A test result that incorrectly indicates disease in someone who is disease-free. Tied to specificity (1 − specificity = false positive rate).
False Negative A test result that incorrectly indicates absence of disease in someone who actually has it. Tied to sensitivity (1 − sensitivity = false negative rate).
Cutpoint (Threshold) The numeric value of a continuous test that separates “positive” from “negative.” Lowering the cutpoint typically raises sensitivity at the expense of specificity.
Methods & Measures
ROC Curve Receiver Operating Characteristic curve: a plot of sensitivity (true positive rate) vs. 1 − specificity (false positive rate) across all possible cutpoints, used to compare tests and choose thresholds.
Area Under the Curve (AUC) A summary of overall test discrimination from the ROC curve, ranging from 0.5 (no better than chance) to 1.0 (perfect). Equivalent to the c-statistic.
Positive Likelihood Ratio (LR+) Sensitivity / (1 − specificity). Indicates how much a positive test increases the odds of disease. Values > 10 strongly rule in disease (Deeks & Altman, 2004).
Negative Likelihood Ratio (LR−) (1 − sensitivity) / specificity. Indicates how much a negative test decreases the odds of disease. Values < 0.1 strongly rule out disease.
Cohen’s Kappa A chance-corrected measure of agreement between two raters or tests on categorical outcomes, κ = (po − pe) ÷ (1 − pe). Ranges from −1 to 1, with 0 for chance agreement, 1 for perfect agreement and negative values when agreement is below chance.
Series Testing Sequential testing in which a positive on one test triggers a second test; disease is declared only if both are positive. Increases overall specificity, decreases overall sensitivity.
Parallel Testing Two tests done simultaneously; disease is declared if either is positive. Increases overall sensitivity, decreases overall specificity.
Screening Programme Concepts
Lead-Time Bias Apparent improvement in survival that arises only because screening detects disease earlier in its course, even when actual time of death is unchanged.
Length Bias Tendency for screening to preferentially detect slow-progressing (longer pre-clinical phase) cases, making screened cases appear to have better outcomes than non-screened cases.
Overdiagnosis Detection of disease that would never have caused symptoms or harm in the patient’s lifetime. Inflates apparent screening benefit and exposes patients to unnecessary treatment (Welch & Black, 2010; Brodersen et al., 2018).
Wilson & Jungner Criteria Ten classic criteria (Wilson & Jungner, 1968, WHO) for evaluating whether a screening programme is appropriate, covering the disease, the test, treatment availability, costs, and ethics. See also Wikipedia: Screening (medicine) and the modern revisit by Andermann et al. (2008).
No matching entries. Try a different search term.
Section 1

Introduction & Test Attributes

⏱ Estimated reading time: 12 minutes

Lesson 5 · HSCI 341

From Populations to Individual Results

Given a single test result, what does it actually tell you about this person?

Section 1 of 4

Introduction & Test Attributes

What a test is, the difference between analytic and diagnostic properties, and how we quantify precision and agreement.

What is a test?

Screening vs. diagnostic tests

Screening test

Applied to apparently healthy populations. Goal: early detection before symptoms appear.

Diagnostic test

Applied to individuals already suspected of disease. Goal: confirm or rule out a condition.

Evaluation principles are the same for both. The clinical context differs.

A critical distinction

Analytic vs. diagnostic sensitivity and specificity

Analytic sensitivity

The lowest concentration of a compound the test can detect. A laboratory property.

Diagnostic sensitivity

The probability that a truly diseased person tests positive. An epidemiological property, covered next.

Analytic specificity: cross-reactivity with other compounds. Diagnostic specificity: correct negative rate. Different quantities entirely.

Test quality

Accuracy and precision

Accurate & precise

Mean near truth; low scatter. The ideal.

Precise, not accurate

Consistent results, but all biased from the truth.

Accurate, not precise

Mean near truth; high variability.

Neither

High scatter and a biased mean.

Coefficient of variation = σ / μ. Lower values indicate greater precision.

Categorical agreement

Cohen's kappa (κ)

Kappa (Cohen, 1960)
\[ \color{#0B7B6B}{\kappa} = \frac{\color{#C2410C}{p_o} - \color{#6D28D9}{p_e}}{1 - \color{#6D28D9}{p_e}} \]
κ chance-corrected agreementpₒ observed agreementpₑ agreement expected by chance

κ ≤ 0: Poor  |  0.01–0.20: Slight  |  0.21–0.40: Fair

0.41–0.60: Moderate  |  0.61–0.80: Substantial  |  0.81–1.00: Almost perfect  (Landis & Koch, 1977)

Weighted kappa

For ordinal scales: near-misses (4 vs. 5) penalised less than large discrepancies (1 vs. 5).

Carry forward

The thread into the next section

  • Screening applies to healthy populations; diagnostic testing to suspected cases. Evaluation principles are shared.
  • Analytic and diagnostic sensitivity are different quantities. Keep them separate.
  • Accuracy, precision, and agreement each ask a version of: how much can you trust the result?

Overview

Measures of disease frequency describe how common a condition is across a population. This lesson applies the same probabilistic vocabulary to a single test given to a single person. Whether you are evaluating a new screening assay, interpreting a clinical result or designing a surveillance algorithm, the same four-cell 2×2 logic appears: a test result that is either positive or negative, set against a true disease state that is either present or absent (Sackett & Haynes, 2002). The lesson builds up from the attributes of the test itself, through sensitivity and specificity and the predictive values that depend on prevalence, to ROC curves and likelihood ratios for tests with continuous results. This section begins by asking what a test is and why screening and diagnosis are different uses of one. It then describes the properties a test has as a measurement, which can be assessed without knowing who truly has the disease: its analytic sensitivity and specificity, its accuracy and precision, and the agreement between repeated or paired results, measured by the coefficient of variation and the limits of agreement for numbers and by Cohen’s kappa for categories.

Learning Objectives

  • 1.1 Distinguish between screening tests and diagnostic tests.
  • 2.1 Define analytic sensitivity and specificity of a test, and distinguish them from diagnostic sensitivity and specificity.
  • 2.2 Explain the difference between accuracy and precision, and describe a test from its repeated results using the four patterns of Figure 5.1.
  • 2.3 Calculate and interpret the coefficient of variation and the limits of agreement for a test that reports a number.
  • 2.4 Calculate and interpret Cohen’s kappa, and explain how prevalence and bias affect it, when weighted kappa is used, and why high agreement does not show that a test is accurate.

1.1 Screening Tests and Diagnostic Tests

A test is any device or procedure designed to detect or quantify a sign, substance, tissue change or body response in an individual. Tests can also be applied at the household or other levels of aggregation. In epidemiology the term “test” extends broadly to include clinical signs, history-taking questions, survey items and post-mortem findings. The characteristics of a test matter in two contexts. In a decision-making context, such as clinical diagnosis, an appropriate test should alter your assessment of the probability that a disease exists and guide the actions that follow, such as further testing, treatment or quarantine. In a research context, understanding the characteristics of a test is essential for knowing how it affects the quality of the data.

Tests are applied for two broad purposes. Screening tests are applied to apparently healthy people to find disease early, and those who screen positive are sent on for a diagnostic work-up. Diagnostic tests are applied to people already considered abnormal, to confirm or classify their disease. Click each card below to see what each kind of test is used for, the condition that makes screening worthwhile, and examples of each.

Screening TestsClick to learn more
Diagnostic TestsClick to learn more

The same test can serve either purpose, so to classify a use, look at whether the people tested have symptoms and at what happens after a positive result, rather than at the name of the test. Despite their different uses, the principles of evaluation and interpretation are the same for screening and diagnostic tests, so the rest of the lesson treats the two together, beginning with the attributes of the test itself.

2.1 Analytic Sensitivity and Specificity

Some properties of a test describe it as a measurement and can be assessed without knowing the true disease status of the people tested. The first two are laboratory properties of an assay. The analytic sensitivity of an assay is the lowest concentration of a chemical compound that it can detect. Its analytic specificity is its capacity to react to only one chemical compound. These are distinctly different from diagnostic (epidemiologic) sensitivity and specificity, which describe how well a test classifies people whose true disease status is known and are the subject of Section 2. A low detection limit does not guarantee a high diagnostic sensitivity, because people tested before the marker has formed, poor samples and reading errors still produce false negatives. When a “sensitivity” is quoted as a concentration, it is an analytic sensitivity; when it is quoted as a proportion of diseased people who test positive, it is a diagnostic sensitivity.

A test that can detect a substance must also measure it well, which raises the questions of accuracy and precision.

2.2 Accuracy and Precision

The laboratory accuracy of a test is its ability to give a true measure of the substance of interest. To be accurate, a test need not always be close to the true value, but if repeat tests are run, their average should be close to the true value. The precision of a test is the consistency of its results. If a test always gives the same value for a sample, whether or not that value is correct, it is said to be precise. Accuracy and precision are therefore separate judgements: accuracy is judged from how far the mean of repeated results lies from the true value, and precision from how widely the results scatter.

Figure 5.1 shows the four combinations as targets, in which the bullseye represents the true value and the dots represent repeated results on the same sample.

Accurate & Precise Inaccurate but Precise Accurate but Imprecise Inaccurate & Imprecise

Figure 5.1. Laboratory accuracy and precision. The bullseye represents the true value.

The first target in Figure 5.1 shows a tight cluster on the bullseye (accurate and precise). The second shows a tight cluster away from it (inaccurate but precise), which is the common trap: a test whose results hardly vary can still be wrong by the same amount every time. The third shows a wide scatter centred on the bullseye (accurate but imprecise), and the fourth a wide scatter away from it (inaccurate and imprecise).

Precision is described in three settings. Repeatability is the variability obtained from repeated testing of the same sample within the same laboratory. Reproducibility is the variability obtained from testing the same sample in different laboratories. Agreement is how well two different tests, or two raters, agree when applied to the same sample. The measures used to put numbers on precision and agreement depend on whether the test reports a number or a category, beginning with tests that report a number.

2.3 Measuring Precision When a Test Reports a Number

When a test reports a number, such as a glucose concentration or a systolic blood pressure, its precision is judged by how closely repeated or paired results agree. Three measures are in common use: the coefficient of variation for repeated results on the same sample, and the concordance correlation coefficient and the limits of agreement for pairs of results from two methods. Expand each panel below to see how each measure is defined and what it reveals.

Coefficient of Variation (CV)▼

The CV describes how much repeated results on the same sample scatter, relative to their average size. It is computed as CV = σ / μ, where σ is the standard deviation among test results on the same sample and μ is their mean. Dividing by the mean matters because a scatter of 5 units is large for a measurement that averages 20 but small for one that averages 500, and the CV expresses the scatter as a fraction of the mean so that the two can be compared fairly. A lower CV indicates greater precision.

Concordance Correlation Coefficient (CCC)▼

The CCC (Lin, 1989) compares two sets of test results on the same samples and better reflects agreement than a Pearson correlation. Picture the paired results plotted against each other. If the two methods agreed perfectly, every point would fall on the equality line, the 45-degree line on which both methods give the same value. The CCC asks how close the points come to that particular line, and it combines three ingredients to do so: the Pearson r, which describes how tightly the points follow some straight line; the location-shift, which describes how far the points sit above or below the equality line; and the scale-shift, which describes whether the points follow a line that is steeper or flatter than the equality line. A CCC of 1 indicates perfect agreement, and any shift or scatter pulls it below 1.

Limits of Agreement (Bland-Altman Plot)▼

A Bland-Altman plot (Bland & Altman, 1986) looks directly at how far apart two methods are for each sample. For every pair of results, the difference between the two methods is plotted against the mean of the pair. Two kinds of line are then drawn across the plot. The mean difference (μd) shows whether one method reads higher than the other on average, which is a systematic bias. The limits of agreement (μd ± 1.96σd) mark the range within which about 95% of the differences fall. Read together, the lines and the points show how large the bias is, how far individual differences stray from it, and whether disagreement grows or shrinks with the size of the measurement.

It is tempting to judge agreement between two methods with a Pearson correlation, but a correlation does not measure agreement. It measures how closely the paired results follow a straight line, any straight line, which is association. Agreement is stricter: it asks whether the results follow the one line on which both methods give the same value. The difference shows up in two common situations. Suppose one method reads 10 units higher than the other on every sample. The points still lie on a perfect straight line, just above the equality line, so the correlation is close to 1 even though the methods never agree. Or suppose one method reads 20% higher at high values. The points again lie on a straight line, this time steeper than the equality line, and the correlation is again close to 1. A constant offset and a different slope both leave the results on a straight line, so a correlation cannot detect either of them. This is why the concordance correlation coefficient adds the location-shift and the scale-shift to the Pearson r. A Bland-Altman plot shows agreement directly. For each pair of results, the difference between the two methods is plotted on the vertical axis against the mean of the pair on the horizontal axis, and horizontal lines mark the mean difference (the systematic bias) and the 95% limits of agreement. The plot shows how large the bias is, how widely individual differences scatter around it, and whether the differences grow as the size of the measurement increases.

The coefficient of variation has no units, so it allows the precision of assays measured on different scales to be compared, but it says nothing about accuracy. The limits of agreement describe the differences to expect for individual samples, so a report should state the direction of the difference (which method minus which) and treat the mean difference as a systematic bias. Two of the measures have simple formulae. Open each tab below to work through one of them: the coefficient of variation (Equation 5.1), applied in Worked Example 5.1 to five repeated measurements of one serum sample and reproduced by Calculator 5.1, and the limits of agreement (Equation 5.2), applied in Worked Example 5.2 to two blood pressure monitors and reproduced by Calculator 5.2. In each calculator, try widening the spread of the results and watch the measure respond.

Coefficient of variation
\[ {\color{#0B7B6B}{CV}} = \dfrac{{\color{#C2410C}{\sigma}}}{{\color{#6D28D9}{\mu}}} \]Eq 5.1
the coefficient of variation equals the standard deviation of repeated results on the same sample divided by the mean of those results.

Worked Example 5.1: Coefficient of Variation

A laboratory runs the same serum sample through a glucose assay five times and obtains 102, 98, 105, 95, and 100 mg/dL. The mean of the five results is μ = 100 mg/dL, and their sample standard deviation (dividing by n − 1 = 4) is σ = 3.81 mg/dL.

CV = 3.81 / 100 = 0.038, or 3.8%. Because the CV divides by the mean, it has no units, which allows the precision of assays measured on different scales to be compared.

Limits of agreement (Bland-Altman)
\[ {\color{#0B7B6B}{\text{limits of agreement}}} = {\color{#C2410C}{\mu_d}} \pm 1.96\,{\color{#6D28D9}{\sigma_d}} \]Eq 5.2
the limits of agreement run from the mean difference between paired results minus 1.96 times the standard deviation of those differences to the mean difference plus the same amount.

Worked Example 5.2: Limits of Agreement

Two automated blood pressure monitors are applied to the same 50 patients. The difference in systolic pressure (monitor A minus monitor B) has a mean of μd = 2.4 mmHg and a standard deviation of σd = 5.0 mmHg.

  • Lower limit = 2.4 − 1.96 × 5.0 = 2.4 − 9.8 = −7.4 mmHg.
  • Upper limit = 2.4 + 1.96 × 5.0 = 2.4 + 9.8 = 12.2 mmHg.

For about 95% of patients, monitor A is expected to read between 7.4 mmHg lower and 12.2 mmHg higher than monitor B. The mean difference of 2.4 mmHg indicates a small systematic bias, and whether a range of this width is acceptable is a clinical judgement.

Worked Example 5.1 gave a CV of 3.8%, showing that the repeated results are consistent with one another, and Worked Example 5.2 showed a small systematic bias between the monitors, with limits whose acceptability is a clinical judgement rather than a statistical one. Both measures describe tests that report a number. Many tests instead place a result in a category, such as positive or negative, and agreement between categorical results needs a measure of its own.

2.4 Agreement for Categorical Results: Cohen’s Kappa

Cohen’s kappa is the usual measure of agreement between two categorical classifications of the same items. It is defined first and applied to two radiologists, and then examined for the ways in which prevalence, bias and ordered categories affect it.

2.4.1 Agreement Beyond Chance

When test results are categorical (dichotomous or ordinal), Cohen’s kappa (κ) measures agreement beyond what would be expected by chance alone (Cohen, 1960). The reason for removing chance is that two raters, or two tests, will agree on some results even if neither is paying attention. If each rater calls about half of the results positive, they would agree on about half of them by guessing alone, much as two people tossing coins would match about half the time. A simple percentage of agreement therefore flatters the raters, because part of it would have happened anyway. Kappa corrects for this. The chance-expected agreement, pe, is worked out from how often each rater uses each category, and kappa then asks how much of the possible agreement beyond chance the raters actually achieved. It compares the observed agreement, po, with the chance agreement (Equation 5.3): the top of the fraction is the agreement the raters achieved beyond chance, and the bottom is the most agreement beyond chance that was possible.

Cohen’s kappa
\[ {\color{#0B7B6B}{\kappa}} = \dfrac{{\color{#C2410C}{p_o}} - {\color{#6D28D9}{p_e}}}{1 - {\color{#6D28D9}{p_e}}} \]Eq 5.3
kappa equals the observed agreement minus the agreement expected by chance, divided by one minus the chance agreement.

Kappa ranges from −1 to 1. A value of 0 means agreement no better than chance, 1 means perfect agreement, and negative values mean agreement below chance, which can happen when two tests tend to pull in opposite directions. Table 5.1 gives the conventional labels of Landis and Koch (1977). They are conventions for describing kappa values, and other authors draw the bands differently, so a report gives the value itself as well as its label.

Table 5.1. Conventional labels for values of kappa (Landis and Koch, 1977).

κ ValueInterpretation
≤ 0Poor agreement
0.01 – 0.20Slight agreement
0.21 – 0.40Fair agreement
0.41 – 0.60Moderate agreement
0.61 – 0.80Substantial agreement
0.81 – 1.00Almost perfect agreement

Worked Example 5.3 applies Equation 5.3 to two radiologists who read the same 100 chest X-rays. Work it in three steps (po, then pe, then κ), and watch how far the chance agreement pulls kappa below the observed agreement.

Worked Example 5.3: Cohen’s Kappa

Two radiologists independently read the same 100 chest X-rays and classify each one as positive or negative for pneumonia.

Radiologist B positiveRadiologist B negativeTotal
Radiologist A positive401050
Radiologist A negative54550
Total4555100
  • Observed agreement: po = (40 + 45) / 100 = 0.85.
  • Agreement expected by chance: radiologist A calls 50% of the films positive and radiologist B calls 45% positive, so pe = (0.50 × 0.45) + (0.50 × 0.55) = 0.225 + 0.275 = 0.50.
  • κ = (0.85 − 0.50) / (1 − 0.50) = 0.35 / 0.50 = 0.70.

The radiologists agree on 85% of the films, but agreement on half of the films (pe = 0.50) would be expected by chance alone. Kappa rescales the agreement beyond chance so that it ranges from −1 to 1, with 0 for chance agreement, 1 for perfect agreement and negative values when agreement is below chance; a value of 0.70 falls in the substantial band of Table 5.1.

The radiologists agree on 85% of the films, but because half of that agreement is expected by chance, kappa is 0.70, substantial agreement in Table 5.1. Calculator 5.3 reproduces the example, so that you can see how changing the cell counts changes kappa.

The same steps work for an agreement table with more than two categories, such as ratings of good, borderline and poor. The observed agreement po is the sum of the cells on the diagonal (where the two raters chose the same category) divided by the total. The chance agreement pe adds, for each category in turn, the proportion of items that rater A placed in it multiplied by the proportion that rater B placed in it. Equation 5.3 then gives kappa as before.

Agreement Is Not Accuracy

Kappa compares two sets of readings with each other, not with the truth. High agreement shows that the readers or tests are consistent, which is precision (reliability). It does not show that either of them is accurate: two readers who make the same mistakes agree closely and are both wrong, like the tight but off-centre cluster of Figure 5.1. Whether a reading is correct can only be judged against a reference standard, through the sensitivity and specificity of Section 2.

2.4.2 How Prevalence and Bias Affect Kappa

Kappa depends on more than how well two raters agree. Because pe is computed from how often each rater uses each category, kappa also depends on the prevalence of the condition among the items rated and on any difference between the raters in how often they call a result positive (Feinstein & Cicchetti, 1990; Byrt, Bishop & Carlin, 1993). Both influences are frequent sources of error in interpreting kappa.

Factors Affecting Kappa

Prevalence: When the condition is very common or very rare, nearly every item falls in the same category for both raters, so most of the agreement sits in one cell of the table. Agreement of that kind is exactly what chance would produce, so pe is close to 1. That leaves very little room for agreement beyond chance (the bottom of Equation 5.3, 1 − pe, is small), and even a little disagreement then produces a low κ. Two tests will therefore have a higher κ when prevalence is moderate (~0.5) than when it is very high or very low, even when they make the same errors. This is the paradox of high observed agreement with a low kappa.

Bias: If one test consistently produces more positive results than the other, κ will be affected. Use McNemar’s χ² test to check whether the two tests classify the same proportion as positive before evaluating agreement. The test uses only the two cells in which the tests disagree (call their counts b and c; in Worked Example 5.3 they are 10 and 5): χ² = (b − c)² ÷ (b + c), on 1 degree of freedom. A value above 3.84 (p < 0.05) suggests that one test calls more results positive than the other; a small value gives no evidence of such a systematic difference.

A small example shows the prevalence effect. Two readers classify 100 films; reader A calls 3 positive and reader B calls 2 positive, and they never call the same film positive. They agree on the 95 films that both call negative, so po = 0.95. Chance agreement is pe = (0.03 × 0.02) + (0.97 × 0.98) = 0.951, so κ = (0.95 − 0.951) ÷ (1 − 0.951) = −0.02. Agreement of 95% is no better than chance here, because almost all of it is agreement on negatives that chance alone would produce. Since nearly every film is negative, two readers who called every film negative without looking would agree just as often. Kappa is not wrong in this case: it correctly reports that the readers never agree on which films are positive. But a reader who saw only the kappa value might wrongly conclude that the readers are careless on every film. Byrt and colleagues (1993) therefore recommended reporting a prevalence index and a bias index alongside kappa, together with the prevalence-adjusted bias-adjusted kappa, PABAK = 2po − 1. PABAK shows what kappa would be if both categories were equally common and the raters used them equally often, so it separates the effect of rarity from the raters' actual consistency. For the two readers, PABAK = 2 × 0.95 − 1 = 0.90, and the gap between 0.90 and −0.02 shows how much of the low kappa comes from the rarity of positives.

2.4.3 Weighted Kappa

Ordinary kappa treats every disagreement as equally serious. For categories with no order, such as positive and negative, that is reasonable. For tests measured on an ordinal scale, it is not: if one rater scores a film 4 and the other scores it 5, they nearly agree, whereas scores of 1 and 5 are far apart. A weighted kappa accounts for this partial agreement by giving pairs of results that are close more credit than pairs that are far apart. It does so by attaching a weight to each cell of the agreement table that reflects how serious that particular disagreement is. Each cell of the k × k agreement table, where k is the number of categories, receives a disagreement weight that is 0 on the diagonal (full agreement) and grows with the distance between the two categories assigned, i and j. Linear weights grow in equal steps with the distance, and quadratic weights grow faster, so they penalise large disagreements much more than small ones: linear weights are |i − j| ÷ (k − 1), and quadratic weights are (i − j)² ÷ (k − 1)². Weighted kappa is 1 minus the ratio of the weighted observed disagreement to the weighted disagreement expected by chance, κw = 1 − Σwijoij ÷ Σwijeij, where oij and eij are the observed and chance-expected proportions in each cell. The idea is the same as in ordinary kappa: the weighted disagreement the raters actually showed is compared with the weighted disagreement chance would produce, so a value of 1 means no disagreement at all and 0 means no better than chance. With quadratic weights it is closely related to the intraclass correlation coefficient (Fleiss & Cohen, 1973). The choice of weights changes the value, so a report states which weights were used. Weighted kappa gives a better reflection of agreement for ordinal data than the unweighted version, which treats every disagreement as equally serious.

This section has distinguished screening tests from diagnostic tests and described the attributes of a test itself: analytic sensitivity and specificity, accuracy and precision, and the coefficient of variation, the limits of agreement and kappa as measures of precision and agreement. None of these attributes shows how often a test classifies a person’s disease status correctly, which requires comparison with a gold standard and is the subject of the next section. The key takeaways and the knowledge check below review the material of this section.

Key Takeaways

  • 1.1 A test is any procedure that detects or measures a sign, substance, tissue change or response in a person, and in epidemiology the word also covers clinical signs, history questions and survey items. Whether a test is being used for screening or for diagnosis depends on who is tested and why. Screening tests are applied to apparently healthy people to find disease early, before it becomes clinical, and those who screen positive are sent on for a fuller diagnostic work-up (mammography for breast cancer is an example). Diagnostic tests are applied to people already considered abnormal, to confirm or classify their disease and guide treatment (a biopsy to confirm cancer, for example). When a question asks you to classify a use, look in the scenario for whether the people tested have symptoms and what happens after a positive result, rather than at the name of the test. The principles you use to evaluate and interpret the test are the same for both uses.
  • 2.1 The words sensitivity and specificity are used in two different ways in this lesson, so it is worth keeping them apart from the start. Analytic sensitivity is a laboratory property of an assay: the lowest concentration of a substance it can detect in a sample. Analytic specificity is the assay’s capacity to react to only the one compound it is meant to detect. Diagnostic (epidemiologic) sensitivity and specificity, which you calculate in the next section, describe something else: how well the test classifies people whose true disease status is known. A low detection limit does not guarantee a high diagnostic sensitivity, because people tested before the marker has formed, poor samples and reading errors still produce false negatives. So when a brochure or a question quotes a “sensitivity” as a concentration (for example in mmol/L), it is describing analytic sensitivity; when it gives the proportion of diseased people who test positive, it is describing diagnostic sensitivity.
  • 2.2 Accuracy and precision describe two different qualities of repeated results on the same sample. A test is accurate when the average of its repeated results is close to the true value, even if single results scatter, and it is precise when its results are consistent with one another, whether or not they are correct. Figure 5.1 shows the four combinations as targets: accurate and precise (a tight cluster on the bullseye), inaccurate but precise (a tight cluster off the bullseye), accurate but imprecise (a wide scatter centred on the bullseye) and inaccurate and imprecise (a wide scatter away from it). When a question gives you the mean and standard deviation of repeated readings and the true value, judge accuracy from how far the mean is from the true value and precision from the size of the standard deviation, and make the two judgements separately. The usual trap is to treat consistency as correctness: a meter whose readings hardly vary can still be wrong by the same amount every time.
  • 2.3 When a test reports a number, such as a glucose concentration or a blood pressure, its precision is described with numbers too. The coefficient of variation, CV = σ / μ, divides the standard deviation of repeated results on the same sample by their mean; in Worked Example 5.1, CV = 3.81 / 100 = 0.038, or 3.8%. Because the CV has no units, it lets you compare the precision of assays measured on different scales, and a lower CV means greater precision, although it tells you nothing about accuracy. The 95% limits of agreement, μd ± 1.96σd, compare two methods used on the same people: in Worked Example 5.2 they run from −7.4 to 12.2 mmHg, so for about 95% of patients monitor A reads between 7.4 mmHg lower and 12.2 mmHg higher than monitor B. When you interpret limits of agreement, state the direction of the difference (which method minus which), treat the mean difference as a systematic bias, and remember that whether the range is narrow enough is a clinical judgement, not a statistical one.
  • 2.4 Cohen’s kappa (κ) measures how well two raters or two tests agree on categorical results once the agreement expected by chance alone is taken out. Work it in three steps: the observed agreement po is the sum of the agreeing (diagonal) cells divided by the total; the chance agreement pe adds, for each category, the proportion rater A placed in it multiplied by the proportion rater B placed in it; and κ = (po − pe) / (1 − pe). In Worked Example 5.3, po = 0.85, pe = 0.50 and κ = 0.70, which Table 5.1 labels substantial agreement, and a good answer reports both the value and its label. Kappa is affected by prevalence (when the condition is very common or very rare, pe is high and kappa can be low despite high observed agreement, which PABAK = 2po − 1 helps to show) and by bias (one rater calling more results positive than the other, which McNemar’s test checks). For ordered categories such as good, borderline and poor, a weighted kappa gives partial credit to near misses, and a report should say which weights were used. Finally, kappa compares two sets of readings with each other, not with the truth, so high agreement shows precision (reliability), not accuracy, which can only be judged against a reference standard.
Knowledge Check: this section

1. A pharmacy offers a free point-of-care HbA1c test to every adult who visits during a health fair, and anyone with a result of 6.5% or higher is advised to see a physician for a fasting glucose test. Separately, a family physician orders a fasting glucose test for a patient who has lost weight and reports thirst and frequent urination. Which statement about these two uses of testing is correct?

This classifies by what the test measures rather than by who is tested and why. The health-fair adults are apparently healthy and are tested to find diabetes early, which makes that a screening use.
Screening applies a test to apparently healthy people to find disease early, while diagnosis confirms or rules out disease in someone already suspected. The principles for evaluating and interpreting a test are the same for both uses.
Referral to a physician is the usual next step after a positive screen, not a sign that the test was diagnostic. The adults at the fair were not suspected of diabetes, so their test is a screening use.
The classification is right, but the conclusion is not: sensitivity and specificity are evaluated and interpreted with the same principles for screening and diagnostic tests. Only the purpose and the people tested differ.
The answer is the second option. The information needed is who is tested and why, not what the test measures: the health-fair test is offered to apparently healthy adults to find diabetes early, and those with a high result go on to further work-up, which is a screening use; the physician tests a person already suspected of diabetes because of symptoms, to confirm or rule it out, which is a diagnostic use. The tempting fourth option is wrong because the principles of evaluation and interpretation are the same for both uses; only the purpose and the people tested differ. The third option mistakes referral for work-up, the usual next step after a positive screen, for diagnosis. (Section 1, 1.1 Screening Tests and Diagnostic Tests, cards “Screening Tests” and “Diagnostic Tests”; Objective 1.1)

2. A laboratory validates a new urine assay for a cannabis metabolite. Its report states that the assay detects concentrations as low as 5 ng/mL and does not react with related compounds. A clinic manager concludes: “So the assay will identify nearly every patient who has used cannabis in the past week.” Which response is correct?

This treats the two meanings of sensitivity as one. A low detection limit is analytic sensitivity; the proportion of recent users who test positive is diagnostic sensitivity, which depends on how much metabolite those people actually have when tested.
The 5 ng/mL figure is the lowest concentration the assay can detect, which is analytic sensitivity, not specificity. Analytic specificity is the assay's capacity to react to only one compound, the second claim in the report.
Analytic sensitivity and specificity describe the assay as a laboratory measurement. Diagnostic sensitivity concerns people, so it must be estimated by testing people of known status against a reference standard.
Analytic sensitivity is the lowest concentration the assay can detect, not a proportion of positive samples. Following the protocol does not make it equal to diagnostic sensitivity, which depends on the metabolite levels in the people tested.
The answer is the third option. Analytic sensitivity is the lowest concentration an assay can detect (here 5 ng/mL), and analytic specificity is its capacity to react to only one compound; both describe the assay as a laboratory measurement. Diagnostic sensitivity is the proportion of people who truly have the condition (here, recent users) who test positive, and it depends on how much metabolite those people actually have in their urine when they are tested, so it must be estimated by comparing the assay with a reference standard in people whose status is known. The tempting first option treats the two meanings of “sensitivity” as the same thing, which is the manager’s error. (Section 1, 2.1 Analytic Sensitivity and Specificity; Objective 2.1)

3. A quality-control sample with a known total cholesterol of 5.00 mmol/L is measured ten times on each of two point-of-care meters. Meter P gives a mean of 5.02 mmol/L with a standard deviation of 0.30 mmol/L. Meter Q gives a mean of 5.60 mmol/L with a standard deviation of 0.04 mmol/L. Which description matches the patterns of Figure 5.1?

Accuracy is how close the mean of repeated results is to the true value, and precision is how little they vary. P is centred near 5.00 but scattered; Q is tightly clustered around 5.60, too high.
This treats a small spread as proof of accuracy. Meter Q's results are consistent, but they cluster around 5.60 mmol/L, well above the true 5.00 mmol/L, and P's mean of 5.02 mmol/L is close to the truth.
This reverses the two properties. Precision is judged from the spread (P's SD of 0.30 is large), and accuracy from how close the mean is to the true value (Q's mean of 5.60 is off by 0.60 mmol/L).
Accuracy is judged by closeness of the mean to the true value, and an arbitrary 15% tolerance is not how the lesson judges it. Meter Q's mean is 0.60 mmol/L too high: precise but inaccurate.
The answer is the first option. Accuracy is judged by how close the mean of repeated results is to the true value, and precision by how much the results vary (the standard deviation). Meter P’s mean (5.02 mmol/L) is close to the true 5.00 mmol/L, but its standard deviation of 0.30 mmol/L shows widely scattered results: accurate but imprecise. Meter Q’s results hardly vary (SD 0.04 mmol/L), but they cluster around 5.60 mmol/L, 0.60 mmol/L too high: precise but inaccurate, the tight off-centre cluster of Figure 5.1. The tempting second option treats a small spread as proof of accuracy, but consistency says nothing about whether the results are centred on the truth. (Section 1, 2.2 Accuracy and Precision, Figure 5.1; Objective 2.2)

4. A wrist blood pressure monitor and a standard arm cuff are used on the same 40 adults. The differences in systolic pressure (wrist minus arm) have a mean of −3.0 mmHg and a standard deviation of 4.0 mmHg. What are the 95% limits of agreement, and what do they mean?

The 95% limits of agreement use 1.96 standard deviations, not one. Using one SD leaves out many individual differences, and the 4 mmHg reading also ignores the mean difference of −3.0 mmHg.
The width is right (1.96 × 4.0 = 7.84), but the limits must be centred on the mean difference of −3.0 mmHg, not on zero. The wrist monitor reads about 3 mmHg lower on average.
The numbers are right but the meaning is wrong. Limits of agreement describe where individual differences fall for about 95% of adults; they are not a confidence interval for the mean difference.
Limits of agreement are μd ± 1.96σd = −3.0 ± 7.84, giving −10.8 to 4.8 mmHg. They describe the range within which about 95% of individual wrist minus arm differences are expected to fall.
The answer is the fourth option. The information needed is the mean difference (μd = −3.0 mmHg) and the standard deviation of the differences (σd = 4.0 mmHg), used in Equation 5.2: limits = μd ± 1.96σd = −3.0 ± (1.96 × 4.0) = −3.0 ± 7.84, which gives −10.8 to 4.8 mmHg. The limits describe individual adults: about 95% of the differences between the two readings are expected to fall in this range, and the negative mean difference shows that the wrist monitor reads about 3 mmHg lower on average. The tempting third option has the right numbers but the wrong meaning, because limits of agreement say where individual differences fall, not where the mean difference lies; the second option leaves out the mean difference, which centres the limits. (Section 1, 2.3 Measuring Precision When a Test Reports a Number, tab “Limits of agreement”, Equation 5.2 and Worked Example 5.2; Objective 2.3)

5. Two school nurses independently check the same 200 children for head lice, with the results below. Calculate Cohen’s kappa and choose the correct interpretation.

Nurse B: liceNurse B: no liceTotal
Nurse A: lice6511
Nurse A: no lice4185189
Total10190200
This reports the observed agreement (0.955) as if it were kappa. Kappa removes the agreement expected by chance, which is very high here because nearly every child is negative.
Kappa = (po − pe) ÷ (1 − pe) = 0.0545 ÷ 0.0995 = 0.55, moderate. When a condition is rare, most agreement is on negatives that chance alone would produce, so kappa falls well below the observed agreement.
This stops after the numerator, po − pe = 0.0545. Kappa divides that by the maximum possible agreement beyond chance, 1 − pe = 0.0995, which gives 0.55.
The value is right but the reason is not. The nurses call almost the same number of children positive (11 and 10), so there is little bias; kappa is low because of the prevalence effect, with lice being rare.
The answer is κ = 0.55, moderate agreement, explained by the rarity of lice. Observed agreement is po = (6 + 185) ÷ 200 = 0.955. Nurse A finds lice in 11 of 200 children (0.055) and nurse B in 10 of 200 (0.05), so chance agreement is pe = (0.055 × 0.05) + (0.945 × 0.95) = 0.00275 + 0.89775 = 0.9005, and κ = (0.955 − 0.9005) ÷ (1 − 0.9005) = 0.0545 ÷ 0.0995 = 0.55, which Table 5.1 labels moderate. When nearly every child is negative, two nurses agree on most children by chance alone, so kappa falls well below the observed agreement (the prevalence effect described in the Factors Affecting Kappa callout; PABAK = 2 × 0.955 − 1 = 0.91 shows what kappa would be if both categories were equally common). The tempting fourth option blames bias, but the nurses call almost the same number of children positive (11 and 10); the first option reports the observed agreement as if it were kappa, and the third stops after the numerator, po − pe. (Section 1, 2.4.1 Agreement Beyond Chance and 2.4.2 How Prevalence and Bias Affect Kappa, Worked Example 5.3 and the callout Factors Affecting Kappa; Objective 2.4)

6. A regional laboratory and a reference laboratory each read the same 150 rapid test strips. They disagree on 16 strips: the regional laboratory alone calls 14 of them positive, and the reference laboratory alone calls the other 2 positive. Using McNemar’s χ² = (b − c)² ÷ (b + c), where b and c are the two discordant counts, with 3.84 as the 5% critical value, which conclusion is correct?

This forgets to square the difference: (14 − 2) ÷ 16 = 0.75, but McNemar's statistic is (14 − 2)² ÷ 16 = 9.0, which is above 3.84.
The statistic is right, but McNemar's test compares the two laboratories with each other, not with the truth. The extra positives may be false positives, and only a reference standard can show which laboratory is more accurate.
McNemar's test uses only the discordant cells: (14 − 2)² ÷ 16 = 9.0, above 3.84. It shows a systematic difference in how often each laboratory calls a strip positive, a bias that affects kappa.
McNemar's test looks for bias, an imbalance between the two discordant cells; it does not measure agreement or show whether kappa is negative. The laboratories agree on 134 of 150 strips.
The answer is the third option. McNemar’s test uses only the two discordant cells, b = 14 and c = 2: χ² = (14 − 2)² ÷ (14 + 2) = 144 ÷ 16 = 9.0, which is above 3.84 (p < 0.05), so one laboratory calls more strips positive than the other. The Factors Affecting Kappa callout recommends this check before evaluating agreement, because such a bias affects kappa. The tempting second option is wrong because the test compares the two laboratories with each other, not with the truth: the extra positives may be false positives, and only a reference standard can tell. The first option forgets to square the difference. (Section 1, 2.4.2 How Prevalence and Bias Affect Kappa, callout Factors Affecting Kappa; Objective 2.4)

✦ Pass the knowledge check with 100% to continue

Section 2

Sensitivity & Specificity

⏱ Estimated reading time: 15 minutes

Section 2 of 4

Sensitivity & Specificity

The 2×2 table, gold standards, and the test properties that travel wherever the test goes.

Reference standard

The gold standard

A gold standard is the reference procedure assumed to be perfectly accurate: all cases classified correctly, no misclassification.

In practice few true gold standards exist. Biological variability and measurement limits mean test evaluation often relies on composite or imperfect references.

The core tool

The 2×2 contingency table

Disease + (D+) Disease − (D−)
Test + (T+) a true positive c false positive
Test − (T−) b false negative d true negative
The key measures

Sensitivity and specificity

Sensitivity
\[ \color{#0B7B6B}{Se} = \frac{\color{#C2410C}{a}}{\color{#C2410C}{a} + \color{#6D28D9}{b}} = \frac{\text{TP}}{\text{all D+}} \]
Se sensitivity (true positive rate)a true positivesb false negatives
Specificity
\[ \color{#0B7B6B}{Sp} = \frac{\color{#C2410C}{d}}{\color{#6D28D9}{c} + \color{#C2410C}{d}} = \frac{\text{TN}}{\text{all D}-} \]
Sp specificity (true negative rate)d true negativesc false positives

SnNOut

High Se: a Negative result rules Out disease.

SpPIn

High Sp: a Positive result rules In disease.

Prevalence effects

True vs. apparent prevalence

Apparent prevalence (Eq 5.6)
\[ \color{#0B7B6B}{AP} = \color{#C2410C}{P} \cdot \color{#6D28D9}{Se} + (1 - \color{#C2410C}{P})(1 - \color{#1D4ED8}{Sp}) \]
AP apparent (test-measured) prevalenceP true prevalenceSe sensitivitySp specificity
Rogan-Gladen correction (1978)
\[ \color{#0B7B6B}{\hat{P}} = \frac{\color{#C2410C}{AP} + \color{#1D4ED8}{Sp} - 1}{\color{#6D28D9}{Se} + \color{#1D4ED8}{Sp} - 1} \]
P̂ corrected true prevalenceAP apparent prevalenceSe sensitivitySp specificity
Carry forward

What sensitivity and specificity cannot tell you

  • Se and Sp are test properties: prevalence does not enter their calculation.
  • They answer the question from the test's viewpoint, not the patient's.
  • To answer “does this positive result mean disease?” you also need disease prevalence. A later section does exactly that.

Overview

Section 1 described the attributes a test has as a measurement. This section turns to the two quantitative properties that capture most of what we care about in a test used on people: sensitivity, its ability to find disease that is truly present, and specificity, its ability to say “no” correctly when disease is truly absent. Both are judged against a gold standard and calculated from a 2×2 table of test result by true disease status. Because each is calculated within one disease group, they are properties of the test, although they can still shift with the kind of disease in the people tested. The section ends by using sensitivity and specificity to move between the true prevalence of a condition and the prevalence a test reports, and by applying all of these measures to the validation of an administrative case definition.

Learning Objectives

  • 3.1 Explain the concept of a gold standard and its role in test evaluation.
  • 3.2 Calculate sensitivity, specificity, false positive fraction, and false negative fraction from a 2×2 table, and interpret a confidence interval reported for sensitivity or specificity.
  • 3.3 Explain why sensitivity and specificity are properties of the test, and why they may still differ between settings (the spectrum effect).
  • 3.4 Distinguish between true prevalence and apparent prevalence.
  • 3.5 Estimate true prevalence from apparent prevalence using the Rogan-Gladen formula.
  • 3.6 Apply sensitivity, specificity, predictive value and the Rogan-Gladen correction to the validation of an administrative case definition.

3.1 The Gold Standard

Sensitivity and specificity are estimated by comparing the results of a test with a reference that is taken to give the true disease status. This kind of comparison was met earlier, when questionnaires were validated in Lesson 3; the recall below links the two, and its retrieval question checks the distinction between reliability and validity evidence.

Recall: Criterion Validity

HSCI 341 Lesson 3, Section 3 (Wording, Structure and Pre-Testing), listed four approaches to validating a questionnaire: comparison with a gold standard, comparison with established methods, repeated administration and method comparison. Comparison with a gold standard gives criterion validity evidence, and sensitivity and specificity, the subject of this section, are that evidence for a categorical measure. Repeated administration to the same respondents is a test-retest design, so it gives reliability evidence: it shows that answers are repeatable, and a repeatable instrument can still measure the wrong thing.

Retrieval question. Classify each of the four Lesson 3 approaches as reliability evidence or validity evidence.

Show the answer▼

Comparison with a gold standard gives criterion validity evidence. Comparison with a well-validated established instrument gives validity evidence (concurrent or convergent), which is only as strong as the established instrument. Repeated administration gives test-retest reliability evidence. Method comparison, which checks whether mail and telephone administration give comparable results, shows agreement between modes, which is reliability (equivalence) evidence unless one mode is treated as the reference.

A gold standard (GS) is a test or procedure that is absolutely accurate: it diagnoses all cases of a specific disease and misdiagnoses none. In reality, very few true gold standards exist. Much of the difficulty comes from biological variability. People do not switch from healthy to “diseased” at the moment of exposure; the disease develops over time, and the time it takes to cross a threshold that any test can detect varies from person to person. Two people infected on the same day may therefore look different on the same test a week later, and even the best reference can miss someone early in the course of the disease. This matters because the estimates of sensitivity and specificity can only be as good as the reference against which they are calculated. Where the reference misclassifies some people, the new test is judged against the wrong answer for them. A test that correctly detects a case the reference missed is then counted as a false positive, so the error of the reference is wrongly charged to the test.

When No Gold Standard Exists

When no true gold standard exists, alternative approaches for estimating sensitivity and specificity are needed, including the use of results from several different tests, repeated testing of selected samples, and latent class models (Hui and Walter, 1980). Latent class models estimate sensitivity and specificity by treating true disease status as an unobserved category, inferred from the pattern of results when several imperfect tests are applied to the same people.

With a gold standard, or the best available reference, in place, each person tested can be classified twice, once by the test and once by the reference, and these two classifications are arranged in a 2×2 table.

3.2 Calculating Sensitivity and Specificity

Sensitivity and specificity, and their complements, the false negative and false positive fractions, are all calculated from the 2×2 table of test result by true disease status. The table is set out first, then the four measures are calculated from it, and finally the confidence intervals that accompany them are interpreted.

3.2.1 The 2×2 Table

Table 5.2 gives the notation used throughout the lesson. Its rows give the true disease status and its columns the test result. Cells a and d hold the correct results and cells b and c the errors; m1 and m0 are the numbers with and without the disease, and n1 and n0 are the numbers who test positive and negative.

Table 5.2. The 2×2 table of disease status by test result, with the notation for its cells and totals.

Test Positive (T+)Test Negative (T−)Total
Disease Positive (D+)a (true positive)b (false negative)m1
Disease Negative (D−)c (false positive)d (true negative)m0
Totaln1n0n

Many published tables, including the worked examples in this lesson, put disease status in the columns instead, and Figure 5.2 shows that layout together with the direction in which each measure is read. The letters keep their meaning (a is always the true positives, b the false negatives, c the false positives and d the true negatives), so the safe habit is to find the disease-status totals first, whichever way the table is drawn.

The 2x2 test table: sensitivity is read down the diseased column, specificity down the non-diseased columnDisease +Disease −Test +Test −atrue positivecfalse positivebfalse negativedtrue negativeSe = a / (a + b)read down the Disease + columnSp = d / (c + d)read down the Disease − columnPredictive values, by contrast, read across the test-result rows.
Figure 5.2. The 2×2 test table. Sensitivity and specificity are computed down the disease-status columns (so they are properties of the test); predictive values are computed across the test-result rows (so they depend on prevalence).

3.2.2 Sensitivity, Specificity and the Error Fractions

Sensitivity (Se) is the proportion of truly diseased people who test positive, and specificity (Sp) is the proportion of truly non-diseased people who test negative. Their complements are the error fractions: the false negative fraction, 1 − Se, is the share of diseased people the test misses, and the false positive fraction, 1 − Sp, is the share of non-diseased people it wrongly calls positive. Click each card below to see the formula for each measure in the notation of Table 5.2, when each kind of error matters most, and the mnemonics for using sensitive and specific tests.

Sensitivity (Se)Click to explore
Specificity (Sp)Click to explore
False Positive FractionClick to explore
False Negative FractionClick to explore

Equation 5.4 and Equation 5.5 give sensitivity and specificity in terms of the cells of the 2×2 table. In each, the denominator is a disease-status total, everyone with the disease for Se and everyone without it for Sp.

Sensitivity
\[ {\color{#0B7B6B}{Se}} = \dfrac{{\color{#C2410C}{a}}}{{\color{#C2410C}{a}} + {\color{#6D28D9}{b}}} \]Eq 5.4
sensitivity equals the true positives divided by everyone who has the disease (the true positives plus the false negatives); the false negative fraction is 1 − Se.
Specificity
\[ {\color{#0B7B6B}{Sp}} = \dfrac{{\color{#BE185D}{d}}}{{\color{#1D4ED8}{c}} + {\color{#BE185D}{d}}} \]Eq 5.5
specificity equals the true negatives divided by everyone who does not have the disease (the false positives plus the true negatives); the false positive fraction is 1 − Sp.

Worked Example 5.4 applies Equations 5.4 and 5.5 to a study of 188 stool samples tested for norovirus with an enzyme immunoassay (EIA). As you read it, check that each measure is divided by a column total of the reference standard, not by a row total of test results.

Worked Example 5.4: Norovirus EIA Data

From a study of 188 stool samples from outbreaks in Ontario, tested with an EIA against a reference standard built from several tests, which this example treats as the gold standard (data from Fisman et al., 2009):

GS+ (D+)GS− (D−)Total
T+71374
T−11103114
Total82106188
  • Se = 71/82 = 86.6% (95% CI: 77.3%, 93.1%)
  • Sp = 103/106 = 97.2% (95% CI: 92.0%, 99.4%)
  • FNF = 1 − 0.866 = 13.4%
  • FPF = 1 − 0.972 = 2.8%

The sensitivity is 71/82, not 71/74: the denominator is the 82 samples with norovirus, not the 74 that tested positive. Dividing by the number who tested positive gives the predictive value of Section 3 instead, which is the most common error in these calculations. Calculator 5.4 reproduces the example, so that you can see how changing the cell counts changes both measures.

3.2.3 Reading a Confidence Interval

Sensitivity and specificity estimated from a study sample are estimates of the test’s true values, and a report gives each with a 95% confidence interval, as in Worked Example 5.4. The interval is the range of values for the true sensitivity that are consistent with the data: if the study were repeated many times, 95% of the intervals calculated in this way would contain the true value. In plain words, the norovirus EIA could detect as few as about 77 or as many as about 93 of every 100 infected people. The width of an interval depends mainly on how many people are in its denominator. Sensitivity is estimated only from the 82 people with the disease, and specificity from the 106 without it, so the sensitivity interval is the wider of the two; a study that enrolled more people with the disease would narrow it. This lesson does not ask you to calculate them.

Because each measure is calculated within one disease group, the next question is what that implies about carrying sensitivity and specificity from one setting to another.

3.3 Sensitivity and Specificity as Properties of the Test

Sensitivity and specificity are calculated within the diseased group and within the non-diseased group, down the columns of Figure 5.2. Sensitivity looks only at people who have the disease and asks what share of them the test detects; specificity looks only at people who do not and asks what share of them the test correctly clears. Neither calculation uses information about how many people are in the other group, so the prevalence of the condition does not enter either of them. If a population had twice as many cases, the sensitivity would stay the same, because twice as many cases would be detected out of twice as many diseased people. For this reason sensitivity and specificity are called properties of the test, and they can be carried from one setting to another more readily than predictive values, which are read across the rows, mix the two groups together and therefore depend on prevalence (Section 3). The same reasoning explains why a highly sensitive test is used to rule disease out after a negative result, and a highly specific test to rule it in after a positive result, as the SnNOut and SpPIn cards in 3.2.2 summarise.

They still depend on who was tested. A test evaluated in people with advanced or severe disease often looks more sensitive than it will be in people with early or mild disease, who may have lower levels of the marker the test detects; this is the spectrum effect. For example, an antibody test evaluated in patients with long-standing infection may miss people tested in the first weeks after infection, before antibodies have formed, so a negative result early in an illness does not rule the disease out. Values estimated in one setting, such as a hospital study, may therefore not hold in a screening population, and before using a published sensitivity you should check whether the people in the evaluation resemble the people you will test.

Design Biases in Diagnostic Accuracy Studies

Published sensitivity and specificity can be inflated by the way a study is designed (Lijmer et al., 1999). Comparing people with severe disease against healthy volunteers, instead of testing the kind of people the test is meant for, produces the spectrum effect described above. Partial verification occurs when only some people, often those who test positive, receive the reference standard, and differential verification when people receive different reference standards depending on their test result. Reading the test while knowing the reference result, or the reverse, also makes the two agree more often. QUADAS-2 (Whiting et al., 2011) is a tool for assessing the risk of bias and the applicability of a diagnostic accuracy study in four domains: patient selection, the index test, the reference standard, and flow and timing. STARD 2015 (Bossuyt et al., 2015) is a reporting checklist that sets out what a diagnostic accuracy study should report, so that readers can judge these problems for themselves.

Sensitivity and specificity describe how a test performs within the diseased and the non-diseased groups. When a test is applied to a whole population, these two quantities and the prevalence together determine the proportion that tests positive.

3.4 True and Apparent Prevalence

A survey that uses an imperfect test counts the people who test positive, which is generally a different number from the people who have the condition. The recall below restates the definition of prevalence from Lesson 4 and its link with incidence and duration, and its retrieval question gives practice with that link.

Recall: Prevalence

Prevalence is the proportion of a population that has a condition at a point in time (point prevalence) or during a period (period prevalence). HSCI 341 Lesson 4, Section 3 (Prevalence, Mortality and Other Frequency Measures), showed that it reflects both how often new cases arise and how long they last: in a steady state, P ÷ (1 − P) = I × D, so P ≈ I × D when the condition is uncommon. HSCI 230 Lesson 3, Section 4, called this the prevalence-duration confound, because factors that prolong a condition can look like causes of it in a cross-sectional study.

Retrieval question. A condition has an incidence rate of 2 per 1,000 person-years and an average duration of 10 years. What is its approximate prevalence in a steady state?

Show the answer▼

I × D = 0.002 × 10 = 0.02, so the prevalence odds are 0.02 and the prevalence is 0.02 ÷ 1.02 = 0.0196, about 2%.

The true prevalence (P) is the actual proportion of the population that has the disease. In Worked Example 5.4, P = 82/188 = 43.6%. The apparent prevalence (AP) is the proportion that tests positive, which includes the false positives and leaves out the false negatives. In Worked Example 5.4, AP = 74/188 = 39.4%. Equation 5.6 expresses the apparent prevalence in terms of the true prevalence and the sensitivity and specificity of the test. Its first term is the true positives, the diseased share of the population multiplied by the sensitivity, and its second term is the false positives, the healthy share multiplied by the false positive fraction.

Apparent prevalence
\[ {\color{#0B7B6B}{AP}} = {\color{#C2410C}{P}}\,{\color{#6D28D9}{Se}} + (1 - {\color{#C2410C}{P}})(1 - {\color{#1D4ED8}{Sp}}) \]Eq 5.6
apparent prevalence equals true prevalence times sensitivity, plus the non-diseased fraction times one minus specificity (the false positives).

Worked Example 5.5 applies Equation 5.6 to the norovirus study. It shows that the formula recovers the apparent prevalence counted from the table, and it separates that figure into its true positive and false positive parts.

Worked Example 5.5: Apparent Prevalence from the Formula

In the norovirus study, the true prevalence is P = 82/188 = 0.4362, and the test has Se = 0.8659 and Sp = 0.9717.

AP = (0.4362 × 0.8659) + (1 − 0.4362) × (1 − 0.9717) = 0.3777 + (0.5638 × 0.0283) = 0.3777 + 0.0160 = 0.3937, or 39.4%.

The formula reproduces the 74/188 = 39.4% counted directly from the table. The first term is the share of the population that is diseased and correctly detected (the true positives), and the second is the share that is healthy but tests positive (the false positives).

Whether the apparent prevalence is too high or too low depends on a tug of war between two kinds of error. The missed cases (false negatives) pull the apparent prevalence down, because people who have the disease are not counted. The false positives push it up, because people without the disease are counted. Here the 11 missed cases outnumber the 3 false positives, so the apparent prevalence (39.4%) is below the true prevalence (43.6%). When a condition is uncommon the balance usually reverses. The healthy group is then very large, and even a small false positive fraction applied to a large group produces many false positives, while the small diseased group can produce only a few missed cases. The false positives then win the tug of war and push the apparent prevalence above the true prevalence. For that reason the proportion testing positive should be called a positivity rate, not a prevalence, unless it has been corrected. Calculator 5.5 reproduces the example; try lowering the true prevalence and watch the apparent prevalence overtake it.

Equation 5.6 runs from the true prevalence to the apparent prevalence. A survey usually needs the reverse.

3.5 Estimating True Prevalence: The Rogan-Gladen Formula

If the Se and Sp of a test are known, the true prevalence can be estimated from the apparent prevalence using the Rogan-Gladen formula (Rogan & Gladen, 1978), Equation 5.7, which is Equation 5.6 solved for P. The logic can be read from its parts. The numerator, AP + Sp − 1, is the apparent prevalence with the false positive fraction (1 − Sp) taken away, which removes the people counted only because the test wrongly called them positive. The denominator, Se + Sp − 1, scales the result back up to allow for the cases the test misses. A perfect test has Se + Sp − 1 = 1 and needs no correction, while a test no better than a coin toss has a denominator of 0 and cannot be corrected at all. Keep every value as a proportion rather than a percentage so that the “− 1” terms make sense, and work out the numerator and the denominator separately before dividing.

True prevalence from apparent prevalence
\[ {\color{#0B7B6B}{P}} = \dfrac{{\color{#C2410C}{AP}} + {\color{#1D4ED8}{Sp}} - 1}{{\color{#6D28D9}{Se}} + {\color{#1D4ED8}{Sp}} - 1} \]Eq 5.7
true prevalence equals apparent prevalence plus specificity minus one, divided by sensitivity plus specificity minus one.

Worked Example 5.6 applies Equation 5.7 to an apparent prevalence of 0.150 measured with a test whose sensitivity is 0.363 and specificity 0.876. Note the check at its end on whether the estimate lies between 0 and 1.

Worked Example 5.6: True Prevalence from the Rogan-Gladen Formula

If AP = 0.150, Se = 0.363, and Sp = 0.876, then:

P = (0.150 + 0.876 − 1) / (0.363 + 0.876 − 1) = 0.026 / 0.239 = 0.109 (10.9%)

Note: Some combinations of Se, Sp, and AP can produce estimates of P outside the range 0–1, indicating that the Se and Sp estimates may not be applicable to the population being studied.

The corrected estimate (10.9%) is lower than the apparent prevalence (15.0%) because, with a specificity of 0.876, the false positives outnumber the cases the test misses; when missed cases outnumber false positives the correction runs the other way. Calculator 5.6 reproduces the example, so that you can see how different values of Se and Sp change the corrected estimate.

These measures, and the Rogan-Gladen correction, apply to any rule that classifies people as cases or non-cases, including a case definition applied to health records.

3.6 Validating an Administrative Case Definition

Surveillance in Canada often counts cases from health administrative records rather than by testing people. The recall below returns to the Canadian Chronic Disease Surveillance System from Lesson 2 and explains why its case definitions can be evaluated as tests.

Recall: The Canadian Chronic Disease Surveillance System

HSCI 341 Lesson 2, Section 1 (Surveillance Systems and Canadian Data Sources), introduced the Canadian Chronic Disease Surveillance System (CCDSS), which applies validated case definitions to health administrative records to estimate the prevalence and incidence of chronic conditions such as diabetes, hypertension and dementia. A case definition of this kind works as a test: each person is classified as a case or a non-case from their records, and the classification can be checked against a reference standard such as chart review.

Retrieval question. Which two kinds of administrative record does the CCDSS draw on?

Show the answer▼

Physician billing claims and hospital discharge abstracts.

Because a case definition works as a test, its validation uses the same 2×2 table, with the chart result as the reference, and the same measures. Worked Example 5.7 validates a diabetes case definition against chart review and then applies the Rogan-Gladen formula (Equation 5.7) to a provincial prevalence estimate. Watch for the step at which the false positives among the large group without diabetes push the administrative estimate above the truth.

Worked Example 5.7: Validating an Administrative Case Definition

A diabetes case definition used with Canadian administrative data counts a person as a case if they have one hospital discharge abstract, or two physician claims within two years, with a diabetes diagnosis. To validate it, a study abstracts the charts of 1,000 adults and compares the case definition with diabetes as documented in the chart, the reference standard. The counts are hypothetical, chosen for teaching.

Chart: diabetesChart: no diabetesTotal
Case definition met8618104
Case definition not met14882896
Total1009001,000
  • Se = 86/100 = 0.86
  • Sp = 882/900 = 0.98
  • PV+ = 86/104 = 0.83

The definition misses 14% of people with diabetes and wrongly includes 2% of people without it, and 17% of the people it counts as cases do not have diabetes according to the chart. The predictive value depends on the prevalence of diabetes in the validation sample (10% here), so it would be lower where diabetes is less common, as the next section shows.

These values change a prevalence estimate. Suppose the case definition, applied to a province's administrative data, gives an apparent prevalence of 8.0%. The Rogan-Gladen formula gives P = (0.080 + 0.98 − 1) ÷ (0.86 + 0.98 − 1) = 0.060 ÷ 0.84 = 0.071, or 7.1%. The administrative estimate is too high by about 0.9 percentage points, because the false positives among the 92.9% of people without diabetes (0.02 × 0.929, or 1.9 percentage points) outnumber the cases the definition misses (0.14 × 0.071, or 1.0 percentage point). A lower specificity would widen the gap, which is why validation studies of administrative case definitions report specificity and predictive value as well as sensitivity.

The example shows the full sequence for a question of this kind: set out the table with the chart as the reference, calculate the measures and put each into plain words, and then explain which kind of error, false positives or missed cases, moves the administrative estimate away from the truth.

This section has defined the gold standard, computed sensitivity and specificity from the 2×2 table, and used them to move between true and apparent prevalence. Worked Example 5.7 also computed a predictive value, which depends on the prevalence in the population tested; the next section develops predictive values in full. The key takeaways and knowledge check below review this material.

Key Takeaways

  • 3.1 Sensitivity and specificity can only be measured by comparing a test with a reference that tells you who truly has the disease, and that reference is called the gold standard. A true gold standard would diagnose every case and misdiagnose none, but very few exist in practice, partly because people do not become detectably diseased the moment they are exposed and the time this takes varies from person to person. When no true gold standard exists, sensitivity and specificity are estimated in other ways, such as combining the results of several tests, retesting selected samples or using latent class models. In a question, identify the reference first (in Worked Example 5.7, for example, it is chart review) and build the 2×2 table around it, because every measure in this section is calculated against it. Keep in mind that the estimates can only be as good as the reference: where the reference itself misclassifies some people, the test is judged against the wrong answer for them.
  • 3.2 Sensitivity is the proportion of people who truly have the disease whom the test calls positive, Se = a / (a + b), and specificity is the proportion of people without the disease whom the test calls negative, Sp = d / (c + d). Their complements are the error fractions: the false negative fraction, 1 − Se, is the share of diseased people the test misses, and the false positive fraction, 1 − Sp, is the share of healthy people it wrongly calls positive. Before you divide, find the reference-standard totals, because the denominator of Se is everyone with the disease and the denominator of Sp is everyone without it, whether the table puts disease status in its rows (Table 5.2) or in its columns (Worked Example 5.4). Dividing by the number who tested positive instead (71/74 rather than 71/82 in the norovirus study) gives the PV+, not the sensitivity, and is the most common mistake. When a confidence interval is reported, read it as the range of true values consistent with the data (the norovirus EIA could detect as few as about 77 or as many as about 93 of every 100 infected people), and expect it to be wider for whichever measure has fewer people in its denominator, here the 82 with the disease rather than the 106 without it.
  • 3.3 Sensitivity and specificity are called properties of the test because each is calculated within one disease group (the diseased for Se, the non-diseased for Sp), so the prevalence of the condition does not enter either one and they carry from one setting to another more readily than predictive values. They still depend on who was tested: a test evaluated in people with advanced or severe disease often looks more sensitive than it will be in people with early or mild disease, which is called the spectrum effect. For example, an antibody test evaluated in long-standing infections may miss people tested in the first weeks, before antibodies form, so a negative result early in an illness does not rule the disease out. Two mnemonics tell you how to use each property: SnNOut (a highly sensitive test with a Negative result rules Out disease) and SpPIn (a highly specific test with a Positive result rules In disease). When a question asks whether published values will hold in a new setting, compare the people in the evaluation (for example a hospital study) with the people you will test (for example a screening population), and name the spectrum effect if their mix of disease differs.
  • 3.4 True prevalence (P) is the proportion of a population that actually has the condition, while apparent prevalence (AP) is the proportion that tests positive. The two differ because the count of positives includes the false positives and leaves out the false negatives: in Worked Example 5.4, P = 82/188 = 43.6% but AP = 74/188 = 39.4%. Equation 5.6, AP = P × Se + (1 − P)(1 − Sp), shows where each part comes from: the first term is the true positives and the second is the false positives among the healthy. When the condition is uncommon, even a small false positive fraction applied to the large healthy group can push the apparent prevalence above the true prevalence, as in the serosurvey question in this section's knowledge check (5.7% against 5%). So in your answers, call the proportion testing positive a positivity rate, not a prevalence, unless it has been corrected for the test’s sensitivity and specificity.
  • 3.5 If you know a test’s sensitivity and specificity, you can work backwards from the apparent prevalence to an estimate of the true prevalence with the Rogan-Gladen formula, P = (AP + Sp − 1) / (Se + Sp − 1). In Worked Example 5.6, an apparent prevalence of 0.150 with Se = 0.363 and Sp = 0.876 gives P = 0.026 / 0.239 = 0.109, or 10.9%. Work out the numerator and the denominator separately before dividing, and keep every value as a proportion rather than a percentage so that the “− 1” terms make sense. Check that your answer lies between 0 and 1: an estimate outside that range tells you that the Se and Sp you used probably do not apply to the population being studied. Then explain the direction of the correction: the true prevalence is lower than the apparent prevalence when false positives outnumber missed cases, and higher when missed cases outnumber false positives.
  • 3.6 A case definition applied to health administrative records, such as the diabetes definition in Worked Example 5.7 (one hospital discharge abstract, or two physician claims within two years), works like a test: it classifies each person as a case or a non-case, and that classification can be checked against a reference standard such as chart review. Its validation therefore uses the same 2×2 table and the same measures: in the worked example, Se = 86/100 = 0.86, Sp = 882/900 = 0.98 and PV+ = 86/104 = 0.83. Put each into plain words: the definition misses 14% of people with diabetes, wrongly includes 2% of people without it, and 17% of the people it counts as cases do not have diabetes according to the chart. Remember that the PV+ depends on the prevalence in the validation sample, so it would be lower where diabetes is less common, while the sensitivity and specificity can be used in the Rogan-Gladen formula to correct a prevalence estimate from administrative data (8.0% corrected to 7.1% in the example). In questions of this kind, set out the table with the chart result as the reference, calculate the measures, and then explain which kind of error, false positives or missed cases, moves the administrative estimate away from the truth.
Knowledge Check: this section

1. A study evaluates a rapid malaria test in returning travellers, using microscopy of a blood film, read by one technologist, as the reference. Microscopy is known to miss some infections in which there are few parasites in the blood. Which statement is correct?

This confuses the reference used in a study with a perfect reference. A gold standard diagnoses every case and misdiagnoses none; microscopy misses low-parasite infections, so the estimates are biased, not exact.
An imperfect reference makes the estimates less trustworthy but does not make them impossible. The lesson lists approaches for when no true gold standard exists: several different tests, repeated testing of selected samples and latent class models.
When the reference misses infections, rapid-test positives in infected travellers are counted as false positives, so the reference's error is charged to the new test. Other approaches exist when no true gold standard is available.
Reading each film twice could improve repeatability (precision), but it cannot make microscopy find parasites it misses. The problem is the accuracy of the reference, not its consistency.
The answer is the third option. A gold standard diagnoses every case and misdiagnoses none, and very few true gold standards exist. When the reference misses some infections, a traveller who is infected, tests positive on the rapid test and is called negative by microscopy is counted as a false positive, so the error of the reference is charged to the new test. The When No Gold Standard Exists callout lists the approaches used when no true gold standard exists: several different tests, repeated testing of selected samples, and latent class models. The tempting first option confuses “the reference used in a study” with “a perfect reference”; the fourth option would improve repeatability (precision), but reading a film twice cannot make microscopy find parasites it misses. (Section 2, 3.1 The Gold Standard, callout When No Gold Standard Exists; Objective 3.1)

2. A rapid test for group A streptococcus (strep throat) is compared with throat culture, the reference standard, in 300 children with a sore throat. The columns give the culture result and the rows the rapid test result. Calculate the sensitivity, specificity, false negative fraction (FNF) and false positive fraction (FPF).

Culture: strepCulture: no strepTotal
Rapid test positive851297
Rapid test negative15188203
Total100200300
These values divide by the row totals (85 ÷ 97 and 188 ÷ 203), which gives the predictive values. Sensitivity and specificity are read down the reference columns, 85 ÷ 100 and 188 ÷ 200.
Se and Sp are right, but the two error fractions are swapped. FNF = 1 − Se = 0.15 is the share of children with strep who are missed, and FPF = 1 − Sp = 0.06.
This swaps sensitivity and specificity. Se is the proportion of children with strep who test positive (85 ÷ 100 = 0.85), and Sp the proportion without strep who test negative (188 ÷ 200 = 0.94).
With culture in the columns, Se = 85 ÷ 100 and Sp = 188 ÷ 200. The error fractions are their complements: FNF = 1 − Se = 0.15 and FPF = 1 − Sp = 0.06.
The answer is Se = 0.85, Sp = 0.94, FNF = 0.15 and FPF = 0.06. The culture result is in the columns, so sensitivity and specificity are read down them: Se = true positives ÷ all children with strep = 85 ÷ 100 = 0.85, and Sp = true negatives ÷ all children without strep = 188 ÷ 200 = 0.94. The false negative fraction is 1 − Se = 15 ÷ 100 = 0.15, and the false positive fraction is 1 − Sp = 12 ÷ 200 = 0.06. The tempting first option divides by the row totals (85 ÷ 97 and 188 ÷ 203), which gives the predictive values, not Se and Sp; the second option swaps the two error fractions. (Section 2, 3.2.2 Sensitivity, Specificity and the Error Fractions, Equations 5.4 and 5.5, Worked Example 5.4; Objective 3.2)

3. The report on the rapid strep test (100 children with strep and 200 without, on throat culture) gives Se = 0.85 (95% CI 0.76 to 0.91) and Sp = 0.94 (95% CI 0.90 to 0.97). Which statement interprets the interval for sensitivity correctly and explains why it is the wider of the two?

A confidence interval gives the values of the true sensitivity consistent with the data. Its width depends mainly on its denominator, and sensitivity rests on 100 children with strep against 200 without for specificity.
The probability of strep given a positive result is PV+, read across the rows, a different quantity from sensitivity. The interval's width depends on the number with strep, not on how many tested positive.
A confidence interval for sensitivity concerns the test's true sensitivity, not the chance that a particular child is classified correctly. Nor are sensitivity intervals always wider; width follows the size of each denominator.
This has the numbers backwards: the study had 100 children with strep and 200 without. The sensitivity interval is wider because its denominator is smaller, not because results varied more.
The answer is the first option. A 95% confidence interval for sensitivity is the range of values for the test’s true sensitivity that are consistent with the data: if the study were repeated many times, 95% of the intervals calculated in this way would contain the true value. Its width depends mainly on the number of people in its denominator; sensitivity is estimated only from the 100 children with strep and specificity from the 200 without, so the sensitivity interval is wider, and enrolling more children with strep would narrow it. The tempting second option describes PV+, the probability of strep given a positive result, which is read across the rows of the table and is a different quantity. The fourth option has the numbers backwards: there were 100 children with strep and 200 without. (Section 2, 3.2.3 Reading a Confidence Interval; Objective 3.2)

4. A rapid influenza antigen test had a sensitivity of 0.90 when it was evaluated in adults admitted to hospital with severe influenza. A community clinic plans to use it for adults with one day of mild symptoms. A colleague says: “Sensitivity is a property of the test, so it will be 0.90 in the clinic too.” Which response is best?

Prevalence does not enter sensitivity, but sensitivity can still change with the spectrum of disease. A test evaluated in severe cases often looks more sensitive than it will be in early or mild cases.
Sensitivity is calculated within people with the disease, so prevalence does not enter it. It still depends on who has the disease: mild, early cases have less of the marker, so the test misses more of them (the spectrum effect).
Lower sensitivity is likely, but the reason given is wrong. A lower prevalence changes the predictive values, not sensitivity; the milder spectrum of illness in the clinic is what is likely to lower it.
This reverses the spectrum effect. People with early or mild illness have lower levels of the marker the test detects, so a test usually misses more of them, not fewer.
The answer is the second option. Sensitivity is calculated within the people who have the disease, so the prevalence does not enter it, which is why it is described as a property of the test. It still depends on who has the disease: a test evaluated in people with severe or advanced disease often looks more sensitive than it will be in people with early or mild disease, who have lower levels of the marker the test detects (the spectrum effect). The tempting third option gives the wrong reason: a lower prevalence changes the predictive values, not the sensitivity, and what is likely to lower the sensitivity here is the milder spectrum of illness. (Section 2, 3.3 Sensitivity and Specificity as Properties of the Test; Objective 3.3)

5. In the rapid strep test study, 100 of the 300 children had strep on throat culture. The rapid test was positive in 97 children: 85 with strep (true positives) and 12 without (false positives). It was negative in 15 children who had strep (false negatives). What are the true prevalence and the apparent prevalence, and why do they differ?

This mixes up the two figures. True prevalence comes from the reference test (100 ÷ 300 = 33.3%) and apparent prevalence from the test positives (97 ÷ 300 = 32.3%), so the test here finds slightly fewer cases.
True prevalence counts every child with strep on culture, including the 15 the test missed, so it is 100 ÷ 300 = 33.3%, not 85 ÷ 300. The apparent prevalence is 32.3%.
True prevalence is 100 ÷ 300 = 33.3% and apparent prevalence 97 ÷ 300 = 32.3%. The apparent figure adds the 12 false positives and loses the 15 false negatives, so it is lower here.
The figures are right but the direction of the reasoning is wrong. A specificity below 1 creates false positives, which push apparent prevalence up; it is lower here because the 15 false negatives outnumber the 12 false positives.
The answer is the third option. True prevalence is the proportion with the condition on the reference test, 100 ÷ 300 = 33.3%; apparent prevalence is the proportion testing positive, 97 ÷ 300 = 32.3%. The apparent prevalence includes the false positives (12) and leaves out the false negatives (15), so here it is lower than the true prevalence because the test misses more children with strep than it wrongly calls positive; when false positives outnumber false negatives, as in most low-prevalence settings, it is higher. The tempting second option counts only the true positives as the true prevalence, but every child with strep counts, including the 15 the test missed. The fourth option gets the direction wrong, because false positives push the apparent prevalence up, not down. (Section 2, 3.4 True and Apparent Prevalence; Objective 3.4)

6. A provincial team validates an asthma case definition for physician billing claims against chart review and finds a sensitivity of 0.75 and a specificity of 0.97. In the province’s claims data, 6.0% of adults meet the definition. Using the Rogan-Gladen formula, P = (AP + Sp − 1) ÷ (Se + Sp − 1), estimate the true prevalence and say why it differs from 6.0%.

Rogan-Gladen gives (0.060 + 0.97 − 1) ÷ (0.75 + 0.97 − 1) = 0.030 ÷ 0.720 = 0.042. When a condition is uncommon, false positives among the many adults without asthma outweigh the missed cases.
Dividing by sensitivity corrects only for missed cases and ignores the false positives. With 3% of the large non-asthma group meeting the definition, the false positives outnumber the missed cases, so the true figure is lower.
This stops after the numerator of the formula, AP + Sp − 1 = 0.030. That figure must still be divided by Se + Sp − 1 = 0.72 to give about 4.2%.
Multiplying by specificity is not the Rogan-Gladen correction. A specificity of 0.97 still means 3% of adults without asthma meet the definition, which is a large error when asthma affects only about 4% of adults.
The answer is about 4.2%. With AP = 0.060, Se = 0.75 and Sp = 0.97, Equation 5.7 gives P = (0.060 + 0.97 − 1) ÷ (0.75 + 0.97 − 1) = 0.030 ÷ 0.720 = 0.042. The claims figure is too high because 3% of the 95.8% of adults without asthma meet the definition (0.03 × 0.958 = 2.9 percentage points of false positives), which outnumbers the cases it misses (0.25 × 0.042 = 1.0 percentage point). The tempting second option corrects only for the missed cases and ignores the false positives, which matter most when a condition is uncommon; the third option stops after the numerator. This is why validation studies of case definitions report specificity and predictive value as well as sensitivity. (Section 2, 3.5 Estimating True Prevalence: The Rogan-Gladen Formula and 3.6 Validating an Administrative Case Definition, Equation 5.7 and Worked Example 5.7; Objectives 3.5 and 3.6)

7. Before a serosurvey, a rapid antibody test for SARS-CoV-2 is evaluated against PCR-confirmed infection and found to have Se = 0.75 and Sp = 0.98. In the serosurvey, 5.65% of adults test positive, and a dashboard reports this as “the proportion of adults who have been infected”. Using the Rogan-Gladen formula, P = (AP + Sp − 1) ÷ (Se + Sp − 1), which statement is correct?

Dividing by sensitivity corrects only for missed infections and ignores the false positives. Rogan-Gladen gives 0.0365 ÷ 0.73 = 0.050, so the dashboard figure is too high, not too low.
Rogan-Gladen gives (0.0565 + 0.98 − 1) ÷ (0.75 + 0.98 − 1) = 0.050. The proportion testing positive is the apparent prevalence, inflated by false positives among the many never infected, so it should be labelled as positivity.
A specificity of 0.98 still produces false positives in 2% of uninfected adults, about 1.9 percentage points here. The proportion testing positive is the apparent prevalence, not the true prevalence, until it is corrected.
This stops after the numerator, AP + Sp − 1 = 0.0365. The Rogan-Gladen formula divides it by Se + Sp − 1 = 0.73, giving about 5.0%.
The answer is the second option. With AP = 0.0565, Se = 0.75 and Sp = 0.98, Equation 5.7 gives P = (0.0565 + 0.98 − 1) ÷ (0.75 + 0.98 − 1) = 0.0365 ÷ 0.73 = 0.050, about 5.0%. The proportion testing positive is the apparent prevalence: it adds the false positives among the 95% never infected (0.02 × 0.95 = 1.9 percentage points) and leaves out the missed infections (0.25 × 0.05 = 1.25 percentage points), so here it is above the true prevalence, and a positivity rate should not be reported as a prevalence until it has been corrected. The tempting first option corrects only for the missed infections and ignores the false positives; the third treats a high specificity as no error at all, and the fourth stops after the numerator. (Section 2, 3.4 True and Apparent Prevalence and 3.5 Estimating True Prevalence: The Rogan-Gladen Formula, Equation 5.7; Objectives 3.4 and 3.5)

8. For the same rapid antibody test, the evaluation reports Se = 0.75 with a 95% confidence interval of 0.68 to 0.81. Most of the infected adults in the evaluation had been admitted to hospital, whereas most infections in the serosurvey population were mild. Which statement is correct?

The probability that a person who tests positive was infected is PV+, not sensitivity. A confidence interval for sensitivity gives the plausible values of the test's true sensitivity.
Prevalence does not enter sensitivity, but it still depends on the spectrum of disease. Evaluation in adults admitted to hospital is likely to overstate sensitivity in a population whose infections were mostly mild.
This reverses the spectrum effect. People with mild infection often make less antibody, so the test is likely to miss more of them and sensitivity in the serosurvey is probably lower.
The interval gives the values of the true sensitivity consistent with the data, in adults like those studied. Because mild infections produce less antibody, sensitivity is probably lower in the serosurvey (the spectrum effect).
The answer is the fourth option. A confidence interval for sensitivity is the range of values for the test’s true sensitivity that are consistent with the data. Prevalence does not enter sensitivity, which is why it is called a property of the test, but sensitivity still depends on the kind of disease in the people tested: a test evaluated in people with severe disease often looks more sensitive than it will be in people with mild disease, who have less of the marker the test detects (the spectrum effect). The tempting second option forgets the spectrum effect; the first option describes PV+, not sensitivity, and the third has the direction of the spectrum effect backwards. (Section 2, 3.2.3 Reading a Confidence Interval and 3.3 Sensitivity and Specificity as Properties of the Test; Objectives 3.2 and 3.3)

✦ Pass the knowledge check with 100% to continue

Section 3

Predictive Values

⏱ Estimated reading time: 12 minutes

Section 3 of 4

Predictive Values

From the test’s perspective to the patient’s: what a result actually means in a given population.

The patient-centred measures

Positive and negative predictive value

PV+ (Eq 5.8)
\[ \color{#0B7B6B}{PV^+} = \frac{\color{#C2410C}{P} \cdot \color{#6D28D9}{Se}}{\color{#C2410C}{P} \cdot \color{#6D28D9}{Se} + (1-\color{#C2410C}{P})(1-\color{#1D4ED8}{Sp})} \]
PV+ positive predictive valueP prevalence (pre-test probability)Se sensitivitySp specificity
PV− (Eq 5.9)
\[ \color{#0B7B6B}{PV^-} = \frac{(1-\color{#C2410C}{P}) \cdot \color{#1D4ED8}{Sp}}{(1-\color{#C2410C}{P}) \cdot \color{#1D4ED8}{Sp} + \color{#C2410C}{P}(1-\color{#6D28D9}{Se})} \]
PV− negative predictive valueP prevalence (pre-test probability)Se sensitivitySp specificity
The prevalence effect

The same test, very different answers

Prevalence PV+ PV−
50%96.9%87.9%
5%61.9%99.3%
0.1%3.0%~100%

Se = 86.6%, Sp = 97.2% held constant. Only prevalence changes.

A public health scenario

HIV universal screening: the arithmetic

PV+ calculation at P = 0.3%
\[ \color{#0B7B6B}{PV^+} = \frac{\color{#C2410C}{0.003} \times \color{#6D28D9}{0.995}}{\color{#C2410C}{0.003} \times \color{#6D28D9}{0.995} + \color{#1D4ED8}{0.997} \times \color{#BE185D}{0.002}} = 60\% \]
0.003 prevalence P0.995 sensitivity Se0.997 1 − P (non-diseased)0.002 1 − Sp (false positive rate)

Se = 99.5%, Sp = 99.8%: an excellent test. Yet 40% of positives are false alarms at low prevalence. Confirmatory testing is arithmetically necessary.

Practical strategies

Increasing PV+ in low-prevalence settings

Target high-risk groups

Higher local prevalence raises PV+ without changing the test itself.

More specific confirmation

Apply a high-Sp test to initial positives. Overall false-positive rate falls sharply.

Series vs. parallel

Series: higher overall Sp, lower Se. Parallel: higher Se, lower Sp. Match strategy to stakes.

Carry forward

What to take into the next section

  • PV+ and PV− are not portable across populations with different prevalence.
  • Prevalence does not enter sensitivity and specificity, so they transfer more readily.
  • A later section asks what happens when we stop forcing continuous results into a binary yes/no.

Overview

Section 2 covered sensitivity and specificity, which are properties of the test itself. This section introduces the predictive values, which tell an individual person what to believe about their disease status given the test result. Predictive values depend on the prevalence of disease in the population tested, which is why the same test can be useful in one setting and misleading in another, and this makes the section the most clinically important in the lesson. It defines the two predictive values, calculates them in three equivalent ways, shows how strongly they depend on prevalence, and describes how the predictive value of a positive result can be raised, including by combining two tests in series or in parallel.

Learning Objectives

  • 4.1 Define predictive value positive (PV+) and predictive value negative (PV−), and explain how each differs from sensitivity and specificity.
  • 4.2 Calculate PV+ and PV− from a 2×2 table, using Bayesian formulas and by counting natural frequencies.
  • 4.3 Explain how prevalence affects predictive values.
  • 4.4 Describe strategies for increasing the predictive value of a positive test.
  • 5.1 Calculate the combined sensitivity and specificity of two tests used in series and in parallel, and choose between the two strategies according to the costs of false positives and false negatives.

4.1 What Predictive Values Tell You

While Se and Sp are characteristics of the test, predictive values tell us how useful the test is for individuals whose disease status is unknown. Once we decide to use a test, we want to know the probability that the individual has, or does not have, the disease given the test result. The predictive value positive (PV+) is the probability that an individual who tests positive actually has the disease, p(D+|T+). The predictive value negative (PV−) is the probability that an individual who tests negative truly does not have the disease, p(D−|T−).

The conditional probabilities run in opposite directions. Sensitivity, p(T+|D+), starts from people known to have the disease and asks how often the test is positive, which is the question from the test’s point of view. PV+, p(D+|T+), starts from people who tested positive and asks how many of them have the disease, which is the question a patient holding the result asks. In the 2×2 table of Figure 5.2, sensitivity and specificity are read down the disease-status columns, while predictive values are read across the test-result rows.

Name the Denominator

The surest way to keep the four measures apart is to name the group in the denominator: everyone with the disease for Se, everyone without it for Sp, everyone who tested positive for PV+ and everyone who tested negative for PV−. When asked what a measure means in plain words, start the sentence from that group.

Because the rows mix diseased and non-diseased people in proportion to the prevalence, predictive values also depend on how common the disease is. Before turning to that dependence, the next step is to calculate them.

4.2 Calculating Predictive Values

Predictive values can be calculated in three ways, which all give the same answer: by dividing cells of a 2×2 table, from a Bayesian formula when only Se, Sp and the prevalence are known, and by counting natural frequencies in a round number of people.

4.2.1 From the 2×2 Table and the Bayesian Formulas

From the 2×2 table, PV+ is the true positives divided by everyone who tested positive, a / n1, and PV− is the true negatives divided by everyone who tested negative, d / n0. Often, however, no 2×2 table is available for the population of interest. A clinician may know only the sensitivity and specificity reported for a test and an estimate of how common the disease is in the patients being tested. Bayes’s theorem gives the same quantities from these three numbers. The formulas look complicated, but they rebuild the 2×2 table in proportions rather than counts. In Equation 5.8 the numerator, prevalence times sensitivity, is the share of all people tested who are true positives. The denominator adds the share who are false positives, the healthy share times the false positive fraction, so that it equals everyone who tests positive. Dividing one by the other is therefore the same as dividing a by n1, and Equation 5.9 does the same for the negatives. Open each tab below to work through one predictive value: each gives the count from the norovirus study, the formula, a worked example that checks the formula against the count (Worked Examples 5.8 and 5.9), and a calculator (Calculators 5.7 and 5.8) in which you can change the prevalence and watch the predictive value respond.

Predictive Value Positive (PV+)

The PV+ is the probability that an individual who tests positive actually has the disease: p(D+|T+) = a / n1.

Positive predictive value
\[ {\color{#0B7B6B}{PV^+}} = \dfrac{{\color{#C2410C}{p(D^+)}}\,{\color{#6D28D9}{Se}}}{{\color{#C2410C}{p(D^+)}}\,{\color{#6D28D9}{Se}} + {\color{#1D4ED8}{p(D^-)}}(1 - {\color{#BE185D}{Sp}})} \]Eq 5.8
the positive predictive value is the true positives, prevalence times sensitivity, divided by all positives (true positives plus false positives, the non-diseased fraction times one minus specificity).

In the norovirus example: PV+ = 71/74 = 95.9% (95% CI: 88.6%, 99.2%)

Worked Example 5.8: PV+ from the Formula

The same answer follows from the formula, using the norovirus study’s prevalence p(D+) = 0.436, Se = 0.866, and Sp = 0.972, so that p(D−) = 1 − 0.436 = 0.564.

PV+ = (0.436 × 0.866) / [(0.436 × 0.866) + 0.564 × (1 − 0.972)] = 0.3776 / (0.3776 + 0.0158) = 0.3776 / 0.3934 = 0.960, or 96.0%.

The numerator is the share of all people tested who are true positives, and the denominator is the share who test positive for any reason. The small difference from 95.9% reflects rounding of the inputs.

Predictive Value Negative (PV−)

The PV− is the probability that an individual who tests negative truly does not have the disease: p(D−|T−) = d / n0.

Negative predictive value
\[ {\color{#0B7B6B}{PV^-}} = \dfrac{{\color{#C2410C}{p(D^-)}}\,{\color{#6D28D9}{Sp}}}{{\color{#C2410C}{p(D^-)}}\,{\color{#6D28D9}{Sp}} + {\color{#1D4ED8}{p(D^+)}}(1 - {\color{#BE185D}{Se}})} \]Eq 5.9
the negative predictive value is the true negatives, the non-diseased fraction times specificity, divided by all negatives (true negatives plus false negatives, the diseased fraction times one minus sensitivity).

In the norovirus example: PV− = 103/114 = 90.4% (95% CI: 83.4%, 95.1%)

Worked Example 5.9: PV− from the Formula

With p(D−) = 0.564, Sp = 0.972, p(D+) = 0.436, and Se = 0.866:

PV− = (0.564 × 0.972) / [(0.564 × 0.972) + 0.436 × (1 − 0.866)] = 0.5482 / (0.5482 + 0.0584) = 0.5482 / 0.6066 = 0.904, or 90.4%.

The numerator is the share of all people tested who are true negatives, and the denominator is the share who test negative for any reason, including the diseased people the test misses.

In the norovirus study both predictive values were high, 95.9% for a positive result and 90.4% for a negative result, and the formulas reproduced the counts. Both, however, were computed at the prevalence in that study, 43.6%.

4.2.2 Counting Natural Frequencies

The formulas can feel abstract, and a reliable check is to walk a round number of people through the test and count them. Split the group by the prevalence, apply the sensitivity to the diseased and the false positive fraction to the healthy, and then divide the true positives by everyone who tested positive. Worked Example 5.10 follows 10,000 people through a test with 99% sensitivity and 95% specificity at a prevalence of 1%. Write down the true positive and false positive counts before you divide, and compare them.

Worked Example 5.10: Predictive Value in Natural Frequencies

The formula can feel abstract, so it helps to walk a whole group of people through the test and simply count. Imagine screening 10,000 people for a disease with a prevalence of 1%, using a strong test with Se = 99% and Sp = 95%.

GroupPeople
Have the disease (1% of 10,000)100
Diseased and test positive (true positives, 99% of 100)99
Diseased and test negative (false negatives)1
Do not have the disease (99% of 10,000)9,900
Healthy and test positive (false positives, 5% of 9,900)495
Healthy and test negative (true negatives)9,405

Now read the positive predictive value straight off the counts. A total of 99 + 495 = 594 people test positive, but only 99 of them truly have the disease, so PV+ = 99 / 594 = 16.7%. About five of every six positive results are false alarms, even though the test is correct 99% of the time in the sick and 95% of the time in the healthy. The reason is arithmetic, not a flaw in the test: the 9,900 healthy people are so numerous that their small 5% error rate yields more false positives (495) than there are true cases in the whole group (100).

Figure 5.3 tells the same story as an interactive animation. Click through its scenes to follow 1,000 people through a test with 95% sensitivity and 95% specificity at a prevalence of 1%, and watch the positive predictive value emerge from the counts of true and false positives.

▸ INTERACTIVE STORY: 1000 PIXEL PEOPLE Open full screen ↗

Watch a 95-95 test scan 1,000 people and see PPV emerge from the math. Next ▶ advances scenes.

Figure 5.3. A 6-scene Bayesian-reasoning visualization: a population of 1,000 with 1% prevalence, a 95%-sensitive 95%-specific test scanning across, the four buckets (TP/FP/FN/TN) populating in real time, and the surprising PPV that follows.

In both Worked Example 5.10 and Figure 5.3, a good test gives a low PV+ because the disease is rare, which leads directly to the role of prevalence.

4.3 How Prevalence Changes Predictive Values

Predictive values depend heavily on the prevalence of disease in the population tested, an application of Bayes's theorem. This is why PV+ and PV− are not good measures of a test’s intrinsic performance: they vary from population to population. Table 5.3 holds the sensitivity (86.6%) and specificity (97.2%) of the norovirus EIA fixed and recomputes both predictive values with Equations 5.8 and 5.9 at three levels of prevalence.

Table 5.3. Predictive values of the norovirus EIA (Se = 86.6%, Sp = 97.2%) at three levels of prevalence.

Prevalence (%)PV+ (%)PV− (%)
5096.987.9
561.999.3
0.13.0100.0

When prevalence drops to 0.1%, the PV+ falls to just 3%, meaning that 97% of positive results are false positives, while the PV− approaches 100%. The reason is arithmetic, not a fault in the test: when the healthy group is very large, even a small false positive fraction produces more false positives than there are true cases, as the counts of Worked Example 5.10 showed. The PV− moves the other way for the mirror-image reason. When the disease is rare, almost everyone who tests negative is healthy simply because almost everyone is healthy, and the few missed cases make up a tiny share of all the negative results. This is a fundamental challenge in screening low-prevalence populations: a negative result is very reassuring, but a positive result is more often wrong than right. Two practical lessons follow. First, when two settings with different prevalence give different PV+ values, the difference should be explained by prevalence, not by a change in the test, because the sensitivity and specificity are the same in both. Second, for the same reason, a predictive value measured in one population should not be carried over to another population with a different prevalence; it has to be recalculated with Equations 5.8 and 5.9 using the new prevalence.

Interactive 5.1 brings the cutoff, the prevalence and the overlap between the test scores of healthy and diseased people together in one simulator. Lower the prevalence slider and notice that sensitivity and specificity stay fixed while PV+ falls and PV− rises; then try the presets for population screening and a symptomatic clinic. Moving the cutoff trades sensitivity against specificity, which Section 4 develops.

🧪 Interactive 5.1: Sensitivity, Specificity, PPV & the Cutoff

This simulator shows the test scores of diseased and healthy people as two overlapping curves, with a cutoff that labels every score to its right as test positive, and fills in the resulting 2×2 table for 10,000 people tested. It is meant to show you that sensitivity and specificity trade off against each other as the cutoff moves, while the predictive values also depend on how common the disease is. As you drag the cutoff line to the right (or move the Cutoff slider), notice that specificity rises and sensitivity falls; as you lower the disease prevalence, sensitivity and specificity stay fixed while PPV (PV+) falls and NPV (PV−) rises. Moving the two mean scores further apart reduces the overlap, so that one cutoff can give high sensitivity and high specificity together.
Distribution of test scores

Drag the dashed cutoff line. Right of the line = test positive.

cutoffTest score02468101214HealthyDiseased
2×2 confusion matrix (per 10,000 tested)
D+D−Total
T+195420203974
T−4659806026
Total2000800010,000
Sensitivity
97.7%
Specificity
74.8%
PPV
49.2%
NPV
99.2%
Presets:
Move the cutoff: see Sn and Sp trade off. Then move prevalence: see PPV/NPV swing while Sn/Sp stay fixed.

If a low prevalence is what drives a low PV+, the remedies follow directly.

4.4 Raising the Predictive Value of a Positive Test

Because a low PV+ comes from false positives among the many healthy people tested, it can be raised by testing a group in which the disease is more common or by reducing the false positives. Three strategies follow from this: targeting high-risk groups, increasing specificity and using more than one test. Click each card below to see how each strategy raises PV+, with an example and the trade-off it involves.

Target High-Risk GroupsClick to explore
Increase SpecificityClick to explore
Use Multiple TestsClick to explore

Each strategy has a cost, such as missed cases in people without a recognised risk factor, a lower sensitivity, or the need for a second test. A common trap is to suggest lowering the cutpoint, which raises Se but lowers Sp and therefore lowers PV+. Worked Example 5.11 applies Equation 5.8 to a proposal for universal HIV screening; notice how low the prevalence is relative to the false positive fraction.

Worked Example 5.11: Universal HIV Screening

A country considers implementing universal HIV screening using a rapid test with Se = 99.5% and Sp = 99.8%. The national HIV prevalence is 0.3%.

PV+ = (0.003 × 0.995) / [(0.003 × 0.995) + (0.997 × 0.002)] = 0.002985 / (0.002985 + 0.001994) = 60.0%

Even with an excellent test (99.5% Se, 99.8% Sp), 40% of positive results in this low-prevalence population would be false positives. This is why confirmatory testing is essential.

Worked Example 5.11 shows why positive results in a low-prevalence population need confirmation, and confirmatory testing is a particular case of combining two tests.

5.1 Combining Two Tests: Series and Parallel

When two tests are used together, the rule for calling a person positive decides what happens to the overall sensitivity and specificity. In series testing, a positive result on the first test triggers the second, and a person is classified as positive only if both tests are positive. In parallel testing, both tests are done at once, and a person is classified as positive if either test is positive. If the errors of the two tests are independent (a person missed by one test is no more likely than anyone else to be missed by the other), the combined values are:

  • In series: combined Se = Se1 × Se2, and combined Sp = 1 − (1 − Sp1)(1 − Sp2).
  • In parallel: combined Se = 1 − (1 − Se1)(1 − Se2), and combined Sp = Sp1 × Sp2.

The logic is the same in both cases. In series, a diseased person must be caught by both tests to be called positive, so sensitivity falls, while a healthy person is called positive only if both tests make a false positive error, so specificity rises. In parallel, a diseased person is missed only if both tests miss them, so sensitivity rises, while a healthy person needs only one false positive to be called positive, so specificity falls. These directions give a quick check on your arithmetic: in series the combined Se should be lower and the combined Sp higher than either test’s alone, and in parallel the reverse. Worked Example 5.12 applies both rules to the same pair of tests and then uses Equation 5.8 to compare the resulting predictive values.

Worked Example 5.12: Two Tests in Series and in Parallel

A first test has Se1 = 0.85 and Sp1 = 0.95, and a second test has Se2 = 0.80 and Sp2 = 0.98. Assume that their errors are independent.

  • Series: Se = 0.85 × 0.80 = 0.68; Sp = 1 − (0.05 × 0.02) = 1 − 0.001 = 0.999.
  • Parallel: Se = 1 − (0.15 × 0.20) = 1 − 0.03 = 0.97; Sp = 0.95 × 0.98 = 0.931.

At a prevalence of 2%, Equation 5.8 gives a PV+ of 0.26 for the first test alone, 0.93 for the two tests in series and 0.22 for the two tests in parallel. Series testing removes most of the false positives but misses almost a third of the cases; parallel testing misses only 3% of the cases but produces more false positives than either test alone.

The choice between the two strategies is a choice about which mistake matters more. Series testing suits screening in a low-prevalence population, where false positives are the main problem and a confirmatory test is applied to the first-stage positives. Parallel testing suits a setting where a missed case would be serious, for example a patient whose treatment would change if the disease were present, and the extra false positives can be resolved by further testing. The Final Knowledge Assessment applies the series and parallel formulas to a two-stage newborn hearing screening programme.

This section has shown that the predictive values of a test depend on the prevalence of the condition as well as on its sensitivity and specificity, so that in a low-prevalence population even a highly specific test can produce mostly false positive results, and that combining tests changes the balance of errors. The key takeaways and knowledge check below review this material, and the next section turns to tests whose results lie on a continuous scale.

Key Takeaways

  • 4.1 Predictive values answer the question a patient actually asks: given my test result, how likely is it that I do, or do not, have the disease? PV+ is the probability of disease given a positive result, p(D+|T+), and PV− is the probability of no disease given a negative result, p(D−|T−). Sensitivity runs the other way, p(T+|D+): it starts from people known to have the disease and asks how often the test is positive, which is the question from the test’s point of view. The surest way to keep the four measures apart is to name the group in the denominator: everyone with the disease for Se, everyone without it for Sp, everyone who tested positive for PV+ and everyone who tested negative for PV−. When a question asks what a measure means in plain words, start your sentence from that group, for example “of the people who test positive on the norovirus EIA, 95.9% truly have norovirus”.
  • 4.2 You can calculate predictive values in three ways, and all three give the same answer. From a 2×2 table, PV+ = a / n1 (true positives divided by everyone who tested positive) and PV− = d / n0 (true negatives divided by everyone who tested negative); in the norovirus study, PV+ = 71/74 = 95.9% and PV− = 103/114 = 90.4%. When you are given only Se, Sp and the prevalence, use the Bayesian formulas (Equations 5.8 and 5.9), in which the numerator is the true positives (or true negatives) and the denominator is everyone with that result, for example PV+ = p(D+)Se / [p(D+)Se + p(D−)(1 − Sp)]. If the formula feels abstract, natural frequencies are a reliable check: take a round number of people such as 10,000, split them by the prevalence, apply Se to the diseased and 1 − Sp to the healthy, and count, as in Worked Example 5.10 (99 true positives and 495 false positives, so PV+ = 99/594 = 16.7%). Whichever method you use, write down the true positive and false positive counts before you divide, because questions often ask for them, and check that they add up to the number who tested positive.
  • 4.3 Predictive values depend on the prevalence of disease in the group tested as well as on Se and Sp, which is why the same test can be very useful in one setting and misleading in another. As prevalence falls, PV+ falls and PV− rises: with the norovirus EIA’s Se of 86.6% and Sp of 97.2%, PV+ is 96.9% at a prevalence of 50% but only 3.0% at 0.1% (Table 5.3). The reason is arithmetic, not a fault in the test: when the healthy group is very large, even a small false positive fraction produces more false positives than there are true cases. So when two settings with different prevalence give different PV+ values, explain the difference by prevalence, and do not conclude that the test performs worse in the low-prevalence setting, because its Se and Sp have not changed. For the same reason, a predictive value from one population should not be carried over to another population with a different prevalence.
  • 4.4 Because a low PV+ comes from false positives among the many healthy people tested, you raise it by reducing those false positives or by testing a group in which the disease is more common. The lesson describes three strategies: target high-risk groups (which raises the pre-test probability), use a more specific test or a cutpoint that raises Sp (usually at some cost in Se), and confirm positive results with a second test in series. Worked Example 5.11 shows why confirmation matters: even an HIV test with Se = 99.5% and Sp = 99.8% has a PV+ of only 60% at a prevalence of 0.3%. A common trap is to suggest lowering the cutpoint, which raises Se but lowers Sp and therefore lowers PV+. When you recommend a strategy, say how it raises PV+ and what it costs, such as more missed cases or the need for a second test.
  • 5.1 When two tests are combined, the rule for calling someone positive decides what happens to sensitivity and specificity. In series, the second test is given only to those positive on the first, and a person is positive only if both tests are positive: combined Se = Se1 × Se2 and combined Sp = 1 − (1 − Sp1)(1 − Sp2), so sensitivity falls while specificity and PV+ rise. In parallel, both tests are done at once and a person is positive if either is positive: combined Se = 1 − (1 − Se1)(1 − Se2) and combined Sp = Sp1 × Sp2, so sensitivity rises and specificity falls. These formulas assume that the two tests make independent errors, and they give a quick check on your arithmetic: in series the combined Se should be lower and the combined Sp higher than either test’s alone, and in parallel the reverse. To choose between them, ask which mistake matters more in the setting: series suits low-prevalence screening, where false positives are the main problem, and parallel suits a setting where a missed case would be serious and the extra positives can be resolved by further testing.
Knowledge Check: this section

1. A news story about a home test for COVID-19 says: “The test has a sensitivity of 0.85, so a person who tests positive has an 85% chance of being infected.” Which correction is right?

This reverses the condition. Sensitivity, p(T+|D+), starts from infected people; the chance of infection given a positive test is PV+, p(D+|T+), which also depends on prevalence.
Specificity is the proportion of uninfected people who test negative, so it does not describe the chance that a positive result is true. That chance is PV+, which depends on Se, Sp and prevalence together.
Sensitivity is read down the diseased column and PV+ across the test-positive row. PV+ also depends on how common infection is among those tested, so it can be far below 0.85.
1 − Se is the false negative fraction, the share of infected people who test negative. It says nothing about the chance that a positive result is true, which is PV+.
The answer is the third option. Sensitivity answers the question from the test’s point of view: of the people who are infected, what proportion does the test call positive? PV+ answers the question that a person holding a positive result asks: of the people who test positive, what proportion are infected? The two conditional probabilities are read in different directions of the 2×2 table (sensitivity down the diseased column, PV+ across the test-positive row), and PV+ also depends on how common the infection is among the people tested, so it can be far below 0.85 when few of them are infected. The tempting first option reverses the condition, which is exactly the error in the news story. (Section 3, 4.1 What Predictive Values Tell You; Objective 4.1)

2. A rapid urine test for chlamydia is compared with a laboratory nucleic acid amplification test (NAAT), the reference standard, in 450 clients of a sexual health clinic. Calculate the PV+ and PV− of the rapid test in this group.

NAAT positiveNAAT negativeTotal
Rapid test positive451257
Rapid test negative5388393
Total50400450
These are the sensitivity (45 ÷ 50) and specificity (388 ÷ 400), read down the reference columns. Predictive values are read across the test-result rows.
Predictive values are read across the rows: PV+ = 45 ÷ 57 = 0.79 true positives among all positive results, and PV− = 388 ÷ 393 = 0.99 true negatives among all negative results.
These divide by the total of 450 clients, which gives the proportions of all clients who are true positives or true negatives. Predictive values divide by the row totals, 57 positive and 393 negative results.
PV+ is right, but 388 ÷ 400 uses the column total of uninfected clients, which is the specificity. PV− divides by all negative results: 388 ÷ 393 = 0.99.
The answer is PV+ = 0.79 and PV− = 0.99. Predictive values are read across the test-result rows: PV+ = true positives ÷ all positive results = 45 ÷ 57 = 0.79, and PV− = true negatives ÷ all negative results = 388 ÷ 393 = 0.99. The tempting first option gives the sensitivity (45 ÷ 50) and the specificity (388 ÷ 400), which are read down the reference columns; the fourth option uses the column total, not the row total, for PV−. (Section 3, 4.2.1 From the 2×2 Table and the Bayesian Formulas, tabs “PV+” and “PV−”, Worked Examples 5.8 and 5.9; Objective 4.2)

3. The rapid chlamydia test has Se = 0.90 and Sp = 0.97, and in the sexual health clinic, where 11% of the clients tested are infected, its PV+ is 0.79. It is then offered in high schools, where about 2% of the students tested are infected. Use PV+ = (P × Se) ÷ [P × Se + (1 − P)(1 − Sp)] to find the PV+ in the schools, and choose the correct explanation of the change.

With P = 0.02, true positives are 0.018 and false positives 0.98 × 0.03 = 0.0294, so PV+ = 0.018 ÷ 0.0474 = 0.38. Se and Sp are unchanged; the large uninfected group supplies more false positives.
Predictive values are not properties of the test alone: they depend on the prevalence in the people tested. At 2% prevalence the false positives outnumber the true positives, and PV+ falls to 0.38.
The value is right, but the reason is not. Prevalence does not enter sensitivity; PV+ falls because the false positives from the many uninfected students outnumber the true positives.
0.018 is only the numerator of the formula, P × Se. PV+ divides it by all positive results, 0.018 + 0.0294, to give 0.38.
The answer is PV+ = 0.38. With P = 0.02, the true positives are 0.02 × 0.90 = 0.018 of the students tested and the false positives are 0.98 × 0.03 = 0.0294, so PV+ = 0.018 ÷ (0.018 + 0.0294) = 0.018 ÷ 0.0474 = 0.38, against 0.79 in the clinic. Sensitivity and specificity are unchanged; what changes is the mix of people tested, and when infection is rare the false positives from the large uninfected group outnumber the true positives, so PV+ falls (and PV− rises). The tempting third option blames a fall in sensitivity, but prevalence does not enter sensitivity; the fourth option stops at the numerator of the formula. (Section 3, 4.2.1 From the 2×2 Table and the Bayesian Formulas, Equation 5.8, and 4.3 How Prevalence Changes Predictive Values, Table 5.3; Objectives 4.2 and 4.3)

4. A health unit screens adults in the general population for hepatitis B with a test that has Se = 0.95 and Sp = 0.98, and finds that most positive results are false. Which change would raise the PV+?

Lowering the cutpoint raises sensitivity but lowers specificity, so false positives increase and PV+ falls further. It works against the goal.
Parallel testing raises sensitivity but lowers specificity, adding false positives and lowering PV+. Series testing, which confirms each positive with a second test, is the combination that raises PV+.
A larger sample makes the estimates more precise but does not change the prevalence, sensitivity or specificity, so the PV+ stays the same.
PV+ depends on prevalence as well as Se and Sp. Testing groups in which hepatitis B is more common raises the pre-test probability, so a larger share of positive results are true.
The answer is to target groups with a higher prevalence. PV+ depends on the prevalence as well as on Se and Sp, so testing people who are more likely to be infected raises the pre-test probability and the share of positive results that are true; using a more specific test, or confirming every positive with a second test in series, would also raise PV+. The tempting first option does the opposite: lowering the cutpoint raises Se but lowers Sp, so false positives increase and PV+ falls. Parallel testing also lowers specificity, and a larger sample makes the estimates more precise but does not change the PV+. (Section 3, 4.4 Raising the Predictive Value of a Positive Test, strategy cards and Worked Example 5.11; Objective 4.4)

5. A first test has Se = 0.90 and Sp = 0.80, and a second test has Se = 0.80 and Sp = 0.95; assume that their errors are independent. In series (positive only if both tests are positive), Se = Se1 × Se2 and Sp = 1 − (1 − Sp1)(1 − Sp2). In parallel (positive if either test is positive), Se = 1 − (1 − Se1)(1 − Se2) and Sp = Sp1 × Sp2. Which set of combined values is correct?

This swaps the two strategies. In series a person with the disease must be caught by both tests, so sensitivity falls to 0.90 × 0.80 = 0.72, while specificity rises to 0.99.
This pairs each strategy's sensitivity with the other strategy's specificity. Series raises specificity to 0.99, and parallel lowers it to 0.80 × 0.95 = 0.76.
Series: Se = 0.90 × 0.80 = 0.72 and Sp = 1 − (0.20 × 0.05) = 0.99. Parallel: Se = 1 − (0.10 × 0.20) = 0.98 and Sp = 0.80 × 0.95 = 0.76. Each strategy trades one property for the other.
Combining tests does not average their properties. Series and parallel testing move sensitivity and specificity in opposite directions, as the given formulas show.
The answer is series Se = 0.72 and Sp = 0.99, parallel Se = 0.98 and Sp = 0.76. Series: Se = 0.90 × 0.80 = 0.72, and Sp = 1 − (0.20 × 0.05) = 1 − 0.01 = 0.99. Parallel: Se = 1 − (0.10 × 0.20) = 1 − 0.02 = 0.98, and Sp = 0.80 × 0.95 = 0.76. In series a person with the disease must be caught by both tests, so sensitivity falls, while a healthy person is called positive only if both tests err, so specificity rises; parallel testing does the reverse. The tempting first option swaps the two strategies, and the second mixes the formulas, pairing each strategy’s sensitivity with the other strategy’s specificity. (Section 3, 5.1 Combining Two Tests: Series and Parallel, Worked Example 5.12; Objective 5.1)

6. Two programmes each plan to combine two tests. Programme 1 screens donated blood for a virus that can be passed on by transfusion: a missed infected unit could infect a recipient, while a false positive only means that a unit is discarded. Programme 2 screens adults in the community for a rare condition, and a positive result leads to an invasive biopsy. Which pairing of strategies is best?

This reverses the trade-offs. Series testing lowers sensitivity, the wrong choice when a missed infected unit is the costly error; parallel testing lowers specificity, the wrong choice when each false positive means a biopsy.
The choice depends on which error is more costly. Parallel testing raises sensitivity, protecting recipients of donated blood; series testing raises specificity and PV+, sparing healthy adults unnecessary biopsies.
It is true that series testing raises PV+, but it also lowers sensitivity. For the blood supply a missed infection is the serious error, so series testing is the wrong trade-off there.
It is true that parallel testing raises sensitivity, but it also lowers specificity. For a rare condition where a positive means a biopsy, the extra false positives make parallel testing the wrong choice.
The answer is parallel for the blood supply and series for community screening. The choice depends on which mistake is more costly. For donated blood a missed infection is the serious error and a false positive costs only a discarded unit, so parallel testing, which raises sensitivity, is preferred. For community screening of a rare condition, false positives are the main problem because each one leads to an invasive biopsy, so series testing, which raises specificity and PV+, is preferred. The tempting third option is true about PV+ but ignores that series testing lowers sensitivity, the wrong trade-off when a missed case is the costly error. (Section 3, 5.1 Combining Two Tests: Series and Parallel, Worked Example 5.12; Objective 5.1)

7. A point-of-care test for antibodies to hepatitis C virus has Se = 0.98 and Sp = 0.99. It is offered at a community health fair, where about 1% of the adults tested have antibodies. For 10,000 adults tested at the fair, which answer gives the correct counts and predictive values, and the correct plain-words meaning of the PV+?

This treats the 2 missed infections as false positives. False positives come from the 9,900 adults without antibodies: 1% of them gives 99, so PV+ = 98 ÷ 197, about 0.50.
The counts and the PV+ are right, but the meaning is reversed. PV+ starts from people who test positive: about half of those who test positive have antibodies. The share of adults with antibodies who test positive is the sensitivity, 0.98.
The counts are right, but predictive values are not equal to specificity and sensitivity. PV+ = 98 ÷ 197, about 0.50, because the false positives from the large antibody-free group equal the true positives.
Natural frequencies give 98 true positives and 99 false positives, so PV+ = 98 ÷ 197, about 0.50, and PV− = 9,801 ÷ 9,803. At 1% prevalence even an excellent test needs a confirmatory test.
The answer is the fourth option. Of 10,000 adults, 100 have antibodies and 9,900 do not. True positives = 0.98 × 100 = 98 (2 are missed), and false positives = (1 − 0.99) × 9,900 = 99, so 197 adults test positive and PV+ = 98 ÷ 197 = 0.50; the true negatives are 9,900 − 99 = 9,801, so PV− = 9,801 ÷ (9,801 + 2) = 0.9998. Because the 9,900 adults without antibodies are so numerous, 1% of them produce as many false positives as there are true positives, so a positive result at the fair is close to a coin toss even though the test is excellent. The tempting second option has the right numbers but reads the PV+ the wrong way round: a predictive value starts from the people who test positive, not from the people with antibodies. The first option treats the 2 missed infections as false positives, and the third confuses predictive values with sensitivity and specificity. (Section 3, 4.1 What Predictive Values Tell You and 4.2.2 Counting Natural Frequencies, Worked Example 5.10; Objectives 4.1, 4.2 and 4.3)

✦ Pass the knowledge check with 100% to continue

Section 4

Cutpoints, ROC Curves & Likelihood Ratios

⏱ Estimated reading time: 15 minutes

Section 4 of 4

Cutpoints, ROC Curves & Likelihood Ratios

What to do when the test gives a number, and how to use it to update the probability of disease.

The overlap problem

Every cutpoint is a trade-off

Healthy Diseased Cutpoint False Negatives False Positives

Shift the cutpoint right: more false negatives, fewer false positives. Shift left: the reverse.

ROC curves

Plotting performance across all cutpoints

1 − Specificity (FPF) Sensitivity chance ROC curve AUC = area under curve
Area under the curve

Summarising discriminatory ability

AUC interpretation (Hanley & McNeil, 1982)
\[ \color{#0B7B6B}{AUC} = P(\color{#C2410C}{\text{score}_{D+}} > \color{#6D28D9}{\text{score}_{D-}}) \]
AUC area under the ROC curvescore (D+) test value of a diseased personscore (D−) test value of a healthy person

0.50: Chance  |  0.50–0.70: Poor  |  0.70–0.80: Acceptable

0.80–0.90: Excellent  |  >0.90: Outstanding

Equivalent to the Mann–Whitney U statistic. Non-parametric confidence intervals available.

Likelihood ratios

Combining Se and Sp into one update factor

LR+ (Eq 5.10)
\[ \color{#0B7B6B}{LR^+} = \frac{\color{#C2410C}{Se}}{1 - \color{#6D28D9}{Sp}} \]
LR+ positive likelihood ratioSe sensitivity (true positive rate)Sp specificity
LR− (Eq 5.11)
\[ \color{#0B7B6B}{LR^-} = \frac{1 - \color{#C2410C}{Se}}{\color{#6D28D9}{Sp}} \]
LR− negative likelihood ratioSe sensitivitySp specificity (true negative rate)

Category-specific LR: \( LR_{cat} = P(\text{result} \mid D+) \;/\; P(\text{result} \mid D-) \). Grades evidence by result magnitude, not just whether it crossed a threshold.

Updating probability

Pre-test to post-test: three steps

Step 1

Pre-test odds = P / (1 − P)

Step 2

Post-test odds = pre-test odds × LR

Step 3

Post-test P = odds / (1 + odds)

Example: pre-test P = 2%, LR = 25.95 → post-test P = 35%. Multiplying the pre-test odds by the LR raises them about 26-fold.

Carry forward

The arc of the lesson

  • ROC curves visualise the Se/Sp trade-off; AUC summarises it.
  • LR+ and LR− combine Se and Sp into a single update factor for each test result.
  • Category-specific LRs grade evidence continuously, not just above or below a threshold.
  • The final review and assessment are just below.

Overview

Sections 1 to 3 judged a test by its accuracy and showed that, in a low-prevalence population, even an accurate test produces many false positives. This section opens by asking when a test should be offered to a whole population as a screening program, a decision that rests on much more than accuracy. It then turns to tests whose results lie on a continuous scale, such as a blood pressure reading, an antibody titre or a probability score, which must be dichotomized at a chosen cutpoint. Every cutpoint trades sensitivity against specificity; the ROC curve and the area under it describe a test across all of its cutpoints, and Youden’s J and the costs of each kind of error guide the choice of one. Finally, likelihood ratios show how much a particular result should change the probability of disease for one person.

Learning Objectives

  • 1.2 Apply the Wilson and Jungner criteria, and their revision by Andermann and colleagues, to decide whether a test should become a screening program, and explain how lead-time bias, length-biased sampling and overdiagnosis distort its evaluation.
  • 6.1 Explain the trade-off between sensitivity and specificity when choosing a cutpoint.
  • 6.2 Describe receiver operating characteristic (ROC) curves and the area under the curve (AUC), and interpret an AUC in plain words.
  • 6.3 Calculate Youden’s J at each of several cutpoints, identify the cutpoint it favours, and choose a lower or higher cutpoint when false negatives or false positives are more costly.
  • 7.1 Define and calculate likelihood ratios for positive and negative test results and for categories of a result.
  • 7.2 Apply likelihood ratios to update pre-test probability to post-test probability.

1.2 From a Test to a Screening Program

Offering a test to a whole population as a screening program raises questions about the condition, its treatment and the program itself, as well as about the test. The principles used to decide whether screening is worthwhile are set out first, then applied to colorectal cancer screening in Canada, and finally three biases that distort the evaluation of screening programs are described.

1.2.1 Principles for Screening

Newborn screening for phenylketonuria (PKU) is a familiar example of screening that works. The recall below summarises it from HSCI 130, and its retrieval question asks which features of PKU made screening worthwhile, which are the features the principles that follow generalise.

Recall: Newborn Screening for PKU

HSCI 130 Lesson 7, Section 4 (Newborn Screening, Precision Medicine, and DTC Testing), presented newborn screening for phenylketonuria (PKU) as public health genetics that works. PKU can be detected from a heel-prick blood spot before any symptoms appear (the Guthrie assay, 1962), and a phenylalanine-restricted diet started early prevents its harm. Every Canadian province now screens newborns for a panel of conditions.

Retrieval question. Which features of PKU made newborn screening for it worthwhile?

Show the answer▼

PKU is serious, it can be detected before symptoms by a simple and acceptable test on a blood spot, and an effective treatment, the restricted diet, works best when it starts early.

The accuracy of a test is one requirement for screening among several. A screening program offers a test to people without symptoms, so it does harm to some of them (false positives, unnecessary follow-up and the treatment of disease that would never have caused illness) as well as good to others. Wilson and Jungner (1968), writing for the World Health Organization, set out ten principles for deciding whether screening for a condition is worthwhile:

  1. The condition sought should be an important health problem.
  2. There should be an accepted treatment for people with recognized disease.
  3. Facilities for diagnosis and treatment should be available.
  4. There should be a recognizable latent or early symptomatic stage.
  5. There should be a suitable test or examination.
  6. The test should be acceptable to the population.
  7. The natural history of the condition, including its development from latent to declared disease, should be adequately understood.
  8. There should be an agreed policy on whom to treat as patients.
  9. The cost of case-finding, including diagnosis and treatment, should be economically balanced in relation to possible expenditure on medical care as a whole.
  10. Case-finding should be a continuing process.

The first eight principles concern the condition, the test and the treatment, and the sensitivity, specificity and predictive values of Sections 2 and 3 are the evidence for the fifth. Andermann and colleagues (2008) reviewed the criteria proposed in the following forty years and synthesized them into a revised set that adds requirements for the program itself: it should respond to a recognized need; define its objectives and its target population at the outset; rest on scientific evidence of effectiveness; integrate education, testing, clinical services and program management; include quality assurance that minimizes the risks of screening; ensure informed choice, confidentiality and respect for autonomy; promote equity and access for the whole target population; plan its evaluation from the outset; and show that its overall benefits outweigh its harms.

1.2.2 Applying the Criteria: Colorectal Cancer Screening

In a scenario, each concern points to a named principle. Worked Example 5.13 applies both sets of criteria to colorectal cancer screening in Canada, first through the principles that concern the condition, the test and the treatment and then through the program-level criteria. As you read it, match each sentence to the principle or criterion it illustrates.

Worked Example 5.13: Applying the Screening Criteria to Colorectal Cancer in Canada

Colorectal cancer is one of the most commonly diagnosed cancers in Canada and a leading cause of cancer death, so it is an important health problem. Most colorectal cancers develop over years from adenomatous polyps, which gives a recognizable early stage, and polyps can be removed during colonoscopy, which gives an accepted treatment. The fecal immunochemical test (FIT), which detects blood in a stool sample collected at home, is a suitable and acceptable test, and the Canadian Task Force on Preventive Health Care (2016) recommends screening adults aged 50 to 74 at average risk with a fecal test every two years or with flexible sigmoidoscopy every ten years. Organized provincial programs offer the test to people in the target age range and recall them when the next test is due, which makes case-finding a continuing process.

The program-level criteria show where such programs succeed or struggle. A positive FIT leads to colonoscopy, so the facilities criterion depends on colonoscopy capacity: long waits after a positive result reduce the benefit of early detection. Quality assurance covers the follow-up of every positive result and the standards for colonoscopy, whose rare complications include bleeding and perforation. Participation varies across population groups, so the equity criterion requires that programs measure who is screened and reach those who are not. Randomized trials of fecal occult blood testing have shown reductions in colorectal cancer mortality, which supports the judgement that benefits outweigh harms for the target age range.

Worked Example 5.13 shows that a suitable test is necessary but not sufficient: long waits for colonoscopy after a positive FIT, for instance, point to the facilities criterion even though the test itself performs well.

1.2.3 Biases in Evaluating Screening

Screening programs are judged by outcomes that three biases can distort, and each of them can make screening look more helpful than it really is. Lead-time bias arises because screening moves the date of diagnosis earlier. Survival is usually counted from diagnosis, so screen-detected cases appear to survive longer simply because the clock started sooner, even when death is not delayed at all. Length-biased sampling arises because screening is done at intervals, such as every two years. A fast-growing cancer may pass through its detectable but symptom-free stage between two screens and be found only when symptoms appear, while a slow-growing cancer stays in that stage for years and is very likely to be caught by a screen. Screen-detected cases are therefore weighted toward slow-growing disease, which has a better prognosis anyway, so they look healthier than symptom-detected cases even if screening does nothing. Overdiagnosis is the extreme form of this problem: the detection of disease that would never have caused symptoms or death in the person's lifetime. Each overdiagnosed person is counted as a screen-detected case who "survived", which flatters the program, although that person gained nothing from the diagnosis and may have been harmed by treatment.

A simple timeline shows lead-time bias. Suppose a cancer would cause symptoms at age 67 and death at age 70, so a person diagnosed after symptoms survives 3 years from diagnosis. If screening finds the same cancer at age 64 and treatment does not change its course, the person still dies at 70, but survival from diagnosis is now 6 years: survival has doubled although no death has been delayed.

All three biases distort a comparison of screen-detected cases with symptom-detected cases, so a sound evaluation avoids that comparison. Lead-time bias lengthens survival measured from diagnosis without changing when people die, so the evaluation compares mortality, the number of deaths from the condition in the population, rather than the survival of cases. Length-biased sampling and overdiagnosis fill the screen-detected group with slow-growing or harmless disease, so the evaluation compares whole groups rather than cases: people randomly assigned to be offered screening against people who are not, counting every case and every death in each group, including cancers that appear between screens. In that comparison, overdiagnosed cases add diagnoses to the screened group but add no deaths, and slow-growing cancers occur in both groups, so neither bias can make screening look better than it is. For these reasons a screening program is evaluated by its effect on mortality from the condition in the whole screened population, ideally in a randomized trial.

The fifth Wilson and Jungner principle asks for a suitable test. For many screening tests, such as FIT, whose result is a concentration, suitability depends on where the line between positive and negative is drawn.

6.1 The Cutpoint and the Trade-off Between Sensitivity and Specificity

Many tests produce results on a continuous or semi-quantitative scale, such as blood urea nitrogen levels, optical density values or enzyme activity. To classify individuals as positive or negative, we select a cutpoint (also called a cut-off or threshold) at or above which a result is called positive. The choice of cutpoint matters because the test values of healthy and diseased people rarely separate cleanly.

The Overlap Problem

In reality, the distributions of test values for healthy and diseased individuals often overlap: some healthy people have fairly high values and some diseased people have fairly low ones. Whatever cutpoint we choose will therefore result in both false positive and false negative results. Raising the cutpoint makes the test harder to pass as positive. Fewer healthy people reach it, so Sp rises (fewer false positives), but some diseased people with modest values now fall below it, so Se falls (more false negatives). Lowering the cutpoint has the opposite effect, catching more of the diseased at the cost of calling more healthy people positive. No cutpoint improves both at once; it can only move errors from one kind to the other.

Figure 5.4 shows the overlap and a cutpoint that divides it. Healthy people whose values fall above the cutpoint are false positives, and diseased people whose values fall below it are false negatives, so a single line cannot separate two groups whose values share a range.

Healthy Diseased Cutpoint False Negatives False Positives Test Value

Figure 5.4. Overlap between healthy and diseased distributions. Moving the cutpoint left or right trades off sensitivity for specificity.

Each cutpoint therefore gives its own pair of sensitivity and specificity values, and a quick check on any table of cutpoints is that sensitivity should fall and specificity rise as the cutpoint goes up. A single graph can display all of these pairs together.

6.2 ROC Curves and the Area Under the Curve

A receiver operating characteristic (ROC) curve plots the Se (y-axis) against the false positive fraction, 1 − Sp (x-axis), computed at a number of different cutpoints (Hanley & McNeil, 1982; see also Wikipedia: ROC curve). Each cutpoint gives one pair of values, so each cutpoint becomes one point on the graph, and joining the points traces the curve. Because the whole range of cutpoints is shown at once, the curve is used both to evaluate the overall performance of a test, without committing to any one cutpoint, and to select a cutpoint. The best possible point is the top-left corner, where sensitivity is 100% and the false positive fraction is 0, so a curve that bows towards the top-left corner belongs to a test that discriminates well. Expand the two panels below to see how to read the curve against its reference lines and how parametric and non-parametric curves are drawn.

Interpreting the ROC Curve▼

The 45° diagonal line represents a test with no discriminating ability (no better than chance). The closer the ROC curve gets to the top-left corner, the better the test discriminates between D+ and D− individuals. The top-left corner represents a test with Se = 100% and Sp = 100%.

Parametric vs. Non-Parametric ROC Curves▼

There are two ways to draw the curve. A non-parametric ROC curve makes no assumptions: it simply uses each observed test value as a cutpoint in turn, plots the Se and (1 − Sp) it produces, and joins the points, which gives a curve made of small steps. A parametric ROC curve instead assumes a shape for the test values in each group, usually that the values of healthy people and the values of diseased people each follow a normal (bell-shaped) distribution, which is called a binormal model. From those two assumed distributions it calculates a smooth curve. The smooth curve is easier to read, but it is only as good as the assumption about the shape of the data. Both approaches can generate 95% confidence intervals.

Each cutpoint is one point on the curve. A very low cutpoint calls almost everyone positive, so both Se and 1 − Sp are close to 1 and its point lies near the top-right corner. Raising the cutpoint lowers both, so a higher (stricter) cutpoint sits further down and to the left on the curve, towards the bottom-left corner, which is reached when the cutpoint is above every test value and no one is called positive.

The area under the curve (AUC) summarises the overall discriminatory ability of the test across all cutpoints. It can be interpreted as the probability that a randomly selected D+ individual has a greater test value than a randomly selected D− individual, which is equivalent to the Mann–Whitney U statistic (Hanley & McNeil, 1982). This interpretation explains the two reference values. A test that carries no information about disease would give the diseased person the higher value only half the time, as a coin toss would, so its AUC is 0.5 and its curve lies along the diagonal. A test whose values for diseased people are all higher than those for healthy people would rank every pair correctly, so its AUC is 1, which is perfect separation. Most real tests fall somewhere in between. Table 5.4 gives conventional labels for ranges of the AUC.

Table 5.4. Conventional interpretation of the area under the ROC curve (AUC).

AUC ValueInterpretation
0.50No discrimination (chance alone)
0.50 – 0.70Poor discrimination
0.70 – 0.80Acceptable discrimination
0.80 – 0.90Excellent discrimination
> 0.90Outstanding discrimination

Interpreting an AUC

State the probability and then the label from Table 5.4: an AUC of 0.85 means that if one diseased and one healthy person are chosen at random, there is an 85% chance that the diseased person has the higher value, which is excellent discrimination. The AUC is not the proportion of people classified correctly, which depends on the cutpoint and on the prevalence, and it does not change when the cutpoint moves.

Interactive 5.2 builds a ROC curve from two overlapping score distributions and reports the sensitivity, the false positive fraction, Youden’s J index (introduced in 6.3) and the AUC. Drag the cutoff and notice that the point moves along the curve while the AUC stays the same; then increase the separation of the distributions or reduce their spread and watch the curve bow towards the upper-left corner as the AUC rises.

📊 Interactive 5.2: ROC Curve Builder

This tool builds a receiver operating characteristic (ROC) curve from the same kind of score distributions as the previous tool: each possible cutoff gives one point, plotting sensitivity against 1 − specificity, and the area under the curve (AUC) summarises how well the test separates the two groups. It is meant to show you that the cutoff decides where you sit on the curve, while the overlap between the two groups decides the shape of the curve itself. As you drag the cutoff line or move its slider, notice that the yellow dot travels along the curve while the AUC stays the same; as you increase the distribution separation or reduce the SD, the curve bows towards the upper-left corner and the AUC climbs towards 1.
Test score distributions

Drag the dashed cutoff line.

02468101214test score
ROC curve

Yellow dot = current cutoff. Diagonal = random-chance reference.

0.00.00.20.20.40.40.60.60.80.81.01.01 − Specificity (False Positive Rate)SensitivityAUC = 0.970
Sensitivity
97.7%
1 − Specificity
25.2%
Youden J
0.725
AUC
0.970
Outstanding discrimination (AUC = 0.970). The curve hugs the upper-left; almost any cutoff is good.

The ROC curve shows every cutpoint at once, but a test in use needs one cutpoint.

6.3 Choosing a Cutpoint with Youden’s J

If sensitivity and specificity are given equal weight, the optimal cutpoint is where Se + Sp is at a maximum, which corresponds to the point on the ROC curve farthest from the 45° line. This maximised value of Se + Sp − 1 is known as Youden’s J index. Subtracting 1 gives J a convenient scale. A test that is no better than chance has Se + Sp = 1 at every cutpoint (calling everyone positive, for example, gives Se = 1 and Sp = 0), so its J is 0, while a perfect test has Se + Sp = 2 and a J of 1. J can therefore be read as how far the test at that cutpoint rises above chance, and it is the quantity Interactive 5.2 reports as you drag the cutoff. Youden’s J can be calculated from any table that gives the sensitivity and specificity at several cutpoints: calculate J at each cutpoint and choose the one where it is largest. When a test reports results in ordered categories, the sensitivity and specificity at a cutpoint are found by counting every category at or above the cutpoint as positive and every category below it as negative, which turns the table into a 2×2 table at that cutpoint. Worked Example 5.14 applies these steps to a blood marker evaluated at four cutpoints; notice how close the two best values of J are, and how the costs of errors then decide the choice.

Worked Example 5.14: Choosing a Cutpoint with Youden’s J

A blood marker is evaluated at four cutpoints, with a value at or above the cutpoint called positive.

Cutpoint (value ≥)20304050
Sensitivity0.980.930.810.62
Specificity0.600.800.910.97
Youden’s J = Se + Sp − 10.580.730.720.59

J is largest (0.73) at a cutpoint of 30, so that is the cutpoint J favours when sensitivity and specificity are given equal weight. Reading the table from left to right shows the trade-off: each higher cutpoint gives a higher specificity and a lower sensitivity. If a missed case were much more costly than a false alarm, for example a serious infection that is easily treated, a lower cutpoint such as 20 would be chosen, accepting more false positives to raise the sensitivity to 0.98. If a false positive led to a risky or expensive follow-up, a higher cutpoint such as 40 would be chosen instead.

Worked Example 5.14 ends by moving away from the Youden point, because giving sensitivity and specificity equal weight is not as neutral as it sounds. The reason is that sensitivity and specificity apply to groups of different sizes. Sensitivity describes errors among the diseased, and specificity describes errors among the healthy. When a disease is uncommon, a one-point gain in specificity prevents far more errors, in absolute numbers, than a one-point gain in sensitivity, simply because there are many more healthy people. Weighting the two percentages equally therefore weights the two kinds of mistake unequally. Put more formally, the Youden point minimises the expected cost of errors only when the cost of a false negative multiplied by the prevalence equals the cost of a false positive multiplied by (1 − prevalence). That means equal costs for each kind of error only when the prevalence is 50%, so that the two groups are the same size. At a prevalence of 10%, the condition reads cost(FN) × 0.10 = cost(FP) × 0.90, so the Youden point quietly treats one missed case as costing as much as nine false positives. J is therefore a starting point rather than the final answer: choose a lower cutpoint to raise sensitivity when a missed case costs more than that, and a higher cutpoint to raise specificity when a false positive leads to a risky or expensive follow-up, stating what the move gains and what it costs. Expand the panel below for a summary of this cost condition and for a second criterion, the point closest to the top-left corner.

Choosing the Optimal Cutpoint▼

The point closest to the top-left corner is a separate criterion from Youden’s J; it agrees with the Youden point when the ROC curve is symmetric and can differ from it when the curve is lopsided. Equal weight on Se and Sp corresponds to equal costs for each false negative and each false positive only when prevalence is 50%. In general, the Youden point minimises the expected cost of errors when the cost of a false negative multiplied by the prevalence equals the cost of a false positive multiplied by (1 − prevalence); at 10% prevalence, for example, it treats one missed case as costing as much as nine false positives. However, if the costs depart from this balance, you might emphasise Se or Sp depending on the clinical context.

The ROC curve, the AUC and Youden’s J describe a test across its cutpoints. A clinician who receives a particular result also needs to know how much that result changes the probability of disease.

7.1 Likelihood Ratios

A likelihood ratio (LR) is the ratio of the probability of a given test result among D+ individuals to the probability of that same result among D− individuals (Deeks & Altman, 2004). Put simply, it asks how much more likely a particular result is in someone with the disease than in someone without it. If a result is common among the diseased and rare among the healthy, seeing it should make us more confident that the person has the disease, and the LR is well above 1. If the result is about equally common in both groups, it tells us little, and the LR is close to 1. If the result is more common among the healthy, it points away from disease, and the LR is below 1. Like sensitivity and specificity, LRs are calculated within the disease groups, so they do not depend on prevalence. They combine the information in Se and Sp into one number, and they convert pre-test odds of disease into post-test odds through Bayes's theorem. There are three kinds: the likelihood ratio for a positive result (Equation 5.10), for a negative result (Equation 5.11) and for a particular category of result (Equation 5.12). Open each tab below to work through one of them. Worked Examples 5.15 and 5.16 apply the first two to the norovirus EIA and Calculators 5.9 and 5.10 reproduce them; Worked Example 5.17 applies the third to a serum marker grouped into three categories, and Calculator 5.11 reproduces it. In the category-specific tab, check that each proportion is divided by the total number of diseased, or of non-diseased, people, not by the total for the category.

Likelihood Ratio for a Positive Test (LR+)

Positive likelihood ratio
\[ {\color{#0B7B6B}{LR^+}} = \dfrac{\color{#C2410C}{Se}}{1 - {\color{#6D28D9}{Sp}}} \]Eq 5.10
the positive likelihood ratio is sensitivity divided by one minus specificity (the true positive rate over the false positive rate).

The LR+ compares the chance of a positive result in the diseased (the sensitivity) with the chance of a positive result in the healthy (the false positive fraction). It is also the factor by which a positive result multiplies the odds of disease: the odds of disease after a positive test result, divided by the odds before the test, equal the LR+. Higher LR+ values therefore mean that a positive test result is more informative for confirming disease.

Worked Example 5.15: LR+

For the norovirus EIA, Se = 71/82 = 0.866 and Sp = 103/106 = 0.9717, so 1 − Sp = 0.0283.

LR+ = 0.866 / 0.0283 = 30.6. A positive EIA result is about 31 times as likely in a person with norovirus as in a person without it.

Likelihood Ratio for a Negative Test (LR−)

Negative likelihood ratio
\[ {\color{#0B7B6B}{LR^-}} = \dfrac{1 - {\color{#C2410C}{Se}}}{\color{#6D28D9}{Sp}} \]Eq 5.11
the negative likelihood ratio is one minus sensitivity divided by specificity (the false negative rate over the true negative rate).

Lower LR− values mean a negative test result is more informative for ruling out disease. An LR− close to 0 is ideal.

Worked Example 5.16: LR−

For the same test, LR− = (1 − 0.866) / 0.9717 = 0.134 / 0.9717 = 0.138. A negative EIA result is about one-seventh as likely in a person with norovirus as in a person without it, so a negative result divides the odds of infection by roughly seven.

Category-Specific LR

Instead of simply classifying results as positive or negative, researchers in diagnostic settings often calculate category-specific LRs based on the actual test value. This uses the actual result rather than just positive/negative, so the strength of evidence is graded by how extreme the value is.

Category-specific likelihood ratio
\[ {\color{#0B7B6B}{LR_{\text{cat}}}} = \dfrac{\color{#C2410C}{P(\text{result}\mid D^+)}}{\color{#6D28D9}{P(\text{result}\mid D^-)}} \]Eq 5.12
a category-specific likelihood ratio is the probability of that result among diseased people divided by its probability among non-diseased people.

Worked Example 5.17: Category-Specific LRs

A serum marker is measured in 100 people with a disease and 200 people without it, and the results are grouped into three categories.

Marker levelD+ (n = 100)D− (n = 200)LRcat
High604(60/100) / (4/200) = 0.60 / 0.02 = 30.0
Intermediate3036(30/100) / (36/200) = 0.30 / 0.18 = 1.67
Low10160(10/100) / (160/200) = 0.10 / 0.80 = 0.125

A high result is 30 times as likely in a person with the disease as in a person without it and strongly supports the diagnosis. An intermediate result changes the odds very little, and a low result reduces them to one-eighth. Collapsing the three categories into a single positive or negative result would discard this gradation.

To interpret any likelihood ratio, use the wording of Worked Example 5.15: a positive EIA result is about 31 times as likely in a person with norovirus as in a person without it. The glossary gives rough guides for judging strength: an LR+ above 10 strongly rules disease in, an LR− below 0.1 strongly rules it out, and values near 1 change the odds very little (Deeks & Altman, 2004). The EIA’s LR+ of 30.6 is well above 10, but its LR− of 0.138 is just above 0.1, so it is better at ruling infection in than out. Worked Example 5.17 adds a further point: the high and low marker categories carry strong evidence that collapsing the results into positive and negative would discard.

A likelihood ratio on its own says how strongly a result points towards or away from disease. To turn it into the probability of disease for a particular person, it has to be combined with what was known before testing.

7.2 From Pre-Test to Post-Test Probability

Likelihood ratios update the assessment of disease probability after a test result in three steps, which Equation 5.13 writes as formulae. First convert the pre-test probability, often the prevalence or a clinical estimate, to pre-test odds. Then multiply the odds by the likelihood ratio for the result obtained. Finally convert the post-test odds back to a probability.

Pre-test to post-test probability
\[ \begin{aligned} \text{odds}_{\text{pre}} &= \dfrac{{\color{#C2410C}{P_{\text{pre}}}}}{1 - {\color{#C2410C}{P_{\text{pre}}}}} \\[4pt] \text{odds}_{\text{post}} &= \text{odds}_{\text{pre}} \times {\color{#6D28D9}{LR}} \\[4pt] {\color{#0B7B6B}{P_{\text{post}}}} &= \dfrac{\text{odds}_{\text{post}}}{1 + \text{odds}_{\text{post}}} \end{aligned} \]Eq 5.13
convert the pre-test probability to odds, multiply those odds by the likelihood ratio for the result obtained, and convert the resulting odds back into the post-test probability.

Convert to Odds First

A likelihood ratio multiplies odds, not probabilities. Multiplying a probability directly by the LR gives a wrong answer, which can even exceed 1, so always convert the probability to odds before multiplying and convert the odds back afterwards.

Worked Example 5.18 applies the three steps of Equation 5.13 to a pre-test probability of 2% and a result whose category-specific LR is 25.95. Follow the three conversions in order.

Worked Example 5.18: Three-Step Process

  1. Convert pre-test probability to pre-test odds: odds = P / (1 − P)
  2. Multiply by the likelihood ratio: post-test odds = pre-test odds × LR
  3. Convert post-test odds back to probability: P = odds / (1 + odds)

Example: Pre-test probability = 2%, test result at a cutpoint where LRcat = 25.95.

  • Pre-test odds = 0.02/0.98 = 0.0204
  • Post-test odds = 0.0204 × 25.95 = 0.5294
  • Post-test probability = 0.5294 / (1 + 0.5294) = 35%

After obtaining the test result, the estimated probability of disease rises from 2% to 35%.

Worked Example 5.18 shows that a strongly positive result can raise a low pre-test probability a long way and still leave the patient more likely than not to be free of disease. Calculator 5.12 applies the three steps to any pre-test probability and likelihood ratio; try a pre-test probability of 30% with the EIA’s LR+ and LR−.

The same three steps apply to a negative result, using LR−. Suppose a patient’s pre-test probability of norovirus is 30%, so the pre-test odds are 0.30 ÷ 0.70 = 0.429. After a positive EIA (LR+ = 30.6, Worked Example 5.15), the post-test odds are 0.429 × 30.6 = 13.1 and the post-test probability is 13.1 ÷ 14.1 = 93%. After a negative EIA (LR− = 0.138, Worked Example 5.16), the post-test odds are 0.429 × 0.138 = 0.059 and the post-test probability is 0.059 ÷ 1.059 = 5.6%. A positive result therefore rules infection in convincingly, while whether a residual probability of about 6% after a negative result is low enough to stop investigating depends on what a missed case would cost.

The three steps also connect likelihood ratios to the predictive values of Section 3. When the pre-test probability is the prevalence of the study sample, the post-test probability after a positive result is the PV+. In the norovirus study the prevalence was 0.436, so the pre-test odds are 0.436 ÷ 0.564 = 0.773; multiplying by LR+ = 30.6 gives post-test odds of 23.7, and 23.7 ÷ 24.7 = 0.96, the PV+ of 95.9% found in 4.2.1. In the same way, the post-test probability after a negative result equals 1 − PV−: the odds become 0.773 × 0.138 = 0.107, a probability of 0.107 ÷ 1.107 = 0.096, which is 1 − 0.904. Both routes apply Bayes’s theorem, but the likelihood ratio route keeps the test (the LR) separate from the setting (the pre-test odds), so it can be applied to any patient’s own pre-test probability rather than only to the prevalence of the study sample.

This section has set the accuracy of a test among the wider criteria for a screening program, shown how the choice of cutpoint determines sensitivity and specificity, how the ROC curve and the AUC summarise a test across all cutpoints, and how likelihood ratios convert a pre-test probability into a post-test probability. The key takeaways and knowledge check below review this material.

Key Takeaways

  • 1.2 An accurate test is only one requirement for a good screening programme, because a programme offers the test to people without symptoms and so harms some of them (false positives, unnecessary follow-up and the treatment of disease that would never have caused illness) while it helps others. The Wilson and Jungner principles ask about the condition (an important health problem with a recognizable early stage and a well-understood natural history), the test (suitable and acceptable to the population), the treatment and the facilities for it, an agreed policy on whom to treat, the cost, and whether case-finding is continuing; Andermann and colleagues added criteria for the programme itself, such as clear objectives, quality assurance, informed choice, equity, a planned evaluation and benefits that outweigh harms. In a scenario, match each concern to the principle it illustrates: long waits for colonoscopy after a positive FIT, for example, point to the facilities criterion (Worked Example 5.13). Be ready to explain the three biases that make screening look better than it is: lead-time bias (earlier diagnosis lengthens survival measured from diagnosis even when death is not delayed), length-biased sampling (screening at intervals tends to find slow-growing disease with a better prognosis) and overdiagnosis (finding disease that would never have caused symptoms or death). Because of these biases, a programme is judged by mortality from the condition in the whole screened population, ideally in a randomized trial, not by the survival of screen-detected cases.
  • 6.1 Many tests give a result on a continuous scale, so someone has to choose a cutpoint at or above which the result is called positive. Because the test values of healthy and diseased people overlap (Figure 5.4), every cutpoint leaves some false positives and some false negatives. Raising the cutpoint calls more healthy people negative, so specificity rises and false positives fall, but it also calls more diseased people with low values negative, so sensitivity falls and false negatives rise; lowering the cutpoint does the opposite. When a question asks why no cutpoint removes both kinds of error, the answer is the overlap: a single line cannot separate two groups whose values share a range. A quick check when you read a table of cutpoints is that sensitivity should fall and specificity should rise as the cutpoint goes up.
  • 6.2 A receiver operating characteristic (ROC) curve shows the whole trade-off at once by plotting sensitivity (y-axis) against the false positive fraction, 1 − Sp (x-axis), at every cutpoint. The 45° diagonal represents a test no better than chance, and the closer the curve bows towards the top-left corner (Se = 100%, Sp = 100%), the better the test discriminates between diseased and healthy people. The area under the curve (AUC) summarises this across all cutpoints: it is the probability that a randomly chosen person with the disease has a higher test value than a randomly chosen person without it. To interpret an AUC in plain words, use that sentence and then add the label from Table 5.4: an AUC of 0.85, for example, means that if one diseased and one healthy person are chosen at random, there is an 85% chance that the diseased person has the higher value, which is excellent discrimination. Remember that the AUC describes the test as a whole and does not change when you move the cutpoint; moving the cutpoint only moves you to a different point on the same curve.
  • 6.3 Youden’s J, J = Se + Sp − 1, is one way to pick a single cutpoint from a table: calculate J at each cutpoint and choose the one where it is largest. In Worked Example 5.14, J is 0.58, 0.73, 0.72 and 0.59 at cutpoints of 20, 30, 40 and 50, so J favours 30. J gives sensitivity and specificity equal weight, so it is a starting point rather than the final answer when the costs of the two kinds of error are clearly unequal. When a missed case is more costly (for example a serious infection that is easily treated), choose a lower cutpoint to raise sensitivity and accept more false positives; when a false positive leads to a risky or expensive follow-up, choose a higher cutpoint to raise specificity. In your answer, name the cutpoint J favours, say which way you would move and why, and quote the change in Se and Sp that the move gains and costs.
  • 7.1 A likelihood ratio (LR) compares how likely a particular test result is in people with the disease with how likely it is in people without it. For a positive result, LR+ = Se / (1 − Sp), and for a negative result, LR− = (1 − Se) / Sp; for the norovirus EIA, LR+ = 0.866 / 0.0283 = 30.6 and LR− = 0.134 / 0.9717 = 0.138. A category-specific LR divides the proportion of diseased people with that result by the proportion of healthy people with it (in Worked Example 5.17, a high marker level gives (60/100) / (4/200) = 30.0), which keeps the information that collapsing results into positive and negative would throw away. To interpret an LR, use the wording of Worked Example 5.15, “a positive EIA result is about 31 times as likely in a person with norovirus as in a person without it”, and judge its strength with the rough guides: an LR+ above 10 strongly rules disease in, an LR− below 0.1 strongly rules it out, and values near 1 change the odds very little. A common slip with category-specific LRs is to divide by the total for that category; each proportion should instead use the total number of diseased, or of healthy, people as its denominator.
  • 7.2 Likelihood ratios let you turn what was believed before testing (the pre-test probability, often the prevalence or a clinical estimate) into a post-test probability for one person, in three steps. First convert the probability to odds, odds = P / (1 − P); then multiply the odds by the LR for the result obtained; then convert back, P = odds / (1 + odds). For a pre-test probability of 30% and the norovirus EIA, the pre-test odds are 0.429, so a positive result gives odds of 13.1 and a probability of 93%, and a negative result gives odds of 0.059 and a probability of 5.6%. The step most often skipped is the conversion: multiplying a probability directly by the LR gives a wrong answer, and can even give a “probability” above 1. When you finish, say whether the result rules disease in or out and whether the remaining probability is low enough to act on, which depends on what a missed case would cost.
Knowledge Check: this section

1. A clinic raises the fasting glucose cutpoint for calling a result positive for diabetes from 6.1 to 7.0 mmol/L. The fasting glucose values of people with and without diabetes overlap. What happens?

This is the effect of lowering the cutpoint. Raising it calls fewer people positive, so specificity rises and false positives fall, while sensitivity falls.
When values overlap, a cutpoint trades one property against the other; a stricter cutpoint raises specificity but lowers sensitivity. Moving the cutpoint changes the balance, not overall accuracy.
Raising the cutpoint moves more people without diabetes below it (higher specificity) and also more people with diabetes (lower sensitivity). Because the values overlap, every cutpoint leaves some of each error.
The direction is right, but the overlap rules out the second claim. A cutpoint high enough to remove every false positive would miss many people with diabetes, so both errors cannot be removed together.
The answer is the third option. Raising the cutpoint means that fewer people are called positive: more people without diabetes now fall below it (fewer false positives, higher specificity), but so do more people with diabetes whose values lie between 6.1 and 7.0 mmol/L (more false negatives, lower sensitivity). Because the values of the two groups overlap, every cutpoint leaves some of each kind of error, and moving it only changes the balance between them. The tempting fourth option has the right direction but ignores the overlap: a cutpoint high enough to remove every false positive would miss many people with diabetes. (Section 4, 6.1 The Cutpoint and the Trade-off Between Sensitivity and Specificity, callout The Overlap Problem; Objective 6.1)

2. Two blood markers are evaluated for the same infection. The ROC curve of marker X lies close to the 45° diagonal (AUC = 0.54). The curve of marker Y bows towards the top-left corner (AUC = 0.88). Which statement is correct?

The AUC is the probability that a randomly chosen person with the disease has a higher value than one without it. 0.88 falls in the excellent band of Table 5.4, and 0.54 is close to the 0.5 of chance.
This reads the AUC as the proportion classified correctly. That proportion depends on the cutpoint chosen and on prevalence, whereas the AUC summarises discrimination across all cutpoints.
Sensitivity changes as the cutpoint moves along the ROC curve; that trade-off is what the curve plots. The AUC is a single summary across all cutpoints, not a fixed sensitivity.
A curve along the 45° diagonal means the test discriminates no better than chance, with an AUC of 0.5. The closer a curve comes to the top-left corner, the better the test, so Y is better.
The answer is the first option. The AUC is the probability that a randomly chosen person with the disease has a higher test value than a randomly chosen person without it; 0.88 lies in the 0.80 to 0.90 band of Table 5.4 (excellent discrimination). A curve on the 45° diagonal, with an AUC of 0.5, discriminates no better than chance, so marker X (0.54) is of little use, and the closer a curve comes to the top-left corner (Se = 100% and Sp = 100%) the better the test. The tempting second option reads the AUC as the proportion of people classified correctly, but that proportion depends on the cutpoint chosen and on the prevalence, whereas the AUC summarises the test across all cutpoints. (Section 4, 6.2 ROC Curves and the Area Under the Curve, Table 5.4; Objective 6.2)

3. A skin meter for newborn jaundice is evaluated against a blood bilirubin test at four cutpoints, with a reading at or above the cutpoint called positive. Missing a baby with severe jaundice can cause permanent brain damage, whereas a false positive leads only to a heel-prick blood test. Which answer is correct?

Cutpoint (reading ≥, µmol/L)150200250300
Sensitivity0.980.930.840.66
Specificity0.600.780.910.97
Youden's J is Se + Sp − 1, not the highest sensitivity; at 150 µmol/L it is 0.98 + 0.60 − 1 = 0.58, the second smallest. J is largest at 250 µmol/L.
J is right, but the direction is wrong. A higher cutpoint raises specificity and lowers sensitivity, missing more babies with severe jaundice, the costly error here.
0.97 is the specificity at 300 µmol/L, not J. There J = 0.66 + 0.97 − 1 = 0.63, smaller than the 0.75 at 250 µmol/L.
J is largest (0.84 + 0.91 − 1 = 0.75) at 250 µmol/L, but J weights Se and Sp equally. When a missed case is far more costly, a lower cutpoint with higher sensitivity (0.93 at 200 µmol/L) is better.
The answer is the fourth option. Youden’s J = Se + Sp − 1 at each cutpoint: 0.98 + 0.60 − 1 = 0.58; 0.93 + 0.78 − 1 = 0.71; 0.84 + 0.91 − 1 = 0.75; 0.66 + 0.97 − 1 = 0.63, so J favours 250 µmol/L. J gives sensitivity and specificity equal weight, but here a missed case is far more costly than a false positive, so a lower cutpoint, with a higher sensitivity (0.93 at 200 µmol/L), is the better choice. The tempting second option finds the right J but moves in the wrong direction: a higher cutpoint raises specificity and would miss more babies with severe jaundice. (Section 4, 6.3 Choosing a Cutpoint with Youden’s J, Worked Example 5.14 and the panel “Choosing the Optimal Cutpoint”; Objective 6.3)

4. A test has Se = 0.80 and Sp = 0.95. Using LR+ = Se ÷ (1 − Sp) and LR− = (1 − Se) ÷ Sp, which answer is correct?

The values are right, but the interpretation is not. A likelihood ratio compares the probability of a result in people with and without the disease; a comparison of disease between test positives and test negatives would depend on prevalence.
LR+ = 0.80 ÷ 0.05 = 16 and LR− = 0.20 ÷ 0.95 = 0.21. A likelihood ratio compares how likely a result is in people with the disease and in people without it.
These are not the likelihood ratio formulas. LR+ divides Se by the false positive fraction, 1 − Sp, and LR− divides the false negative fraction, 1 − Se, by Sp.
This swaps the two ratios. A positive result raises the odds of disease, so LR+ is above 1 (16); a negative result lowers them, so LR− is below 1 (0.21).
The answer is the second option. LR+ = 0.80 ÷ (1 − 0.95) = 0.80 ÷ 0.05 = 16, and LR− = (1 − 0.80) ÷ 0.95 = 0.20 ÷ 0.95 = 0.21. A likelihood ratio compares the probability of a result in people with the disease with its probability in people without it, so a positive result is 16 times as likely in someone with the disease, and it multiplies the pre-test odds of disease by 16. The tempting first option compares people who test positive with people who test negative, which is a statement about the probability of disease given the result; that would depend on the prevalence and is not what a likelihood ratio measures. (Section 4, 7.1 Likelihood Ratios, tabs “LR+” and “LR−”, Equations 5.10 and 5.11, Worked Examples 5.15 and 5.16; Objective 7.1)

5. For the test in the previous question (LR+ = 16, LR− = 0.21), a physician judges a patient’s probability of disease to be 20% before testing. What is the probability of disease after a positive result and after a negative result?

This treats the probability of 0.20 as odds (0.20 × 16 = 3.2). A likelihood ratio multiplies odds, so 0.20 is first converted to odds of 0.25, giving 4.0 and then 80%.
A likelihood ratio multiplies the odds, not the probability, so the result never exceeds a probability of 1. Odds of 0.25 × 16 = 4.0 convert back to 4.0 ÷ 5.0 = 80%.
Pre-test odds are 0.20 ÷ 0.80 = 0.25. Multiplying by 16 gives 4.0 (80%), and by 0.21 gives 0.0525 (5.0%), converting back with odds ÷ (1 + odds).
The positive result is right, but 21% is the LR− itself reported as a probability. Pre-test odds of 0.25 × 0.21 = 0.0525 convert back to about 5.0%.
The answer is 80% and 5.0%. Step 1: pre-test odds = 0.20 ÷ (1 − 0.20) = 0.25. Step 2: post-test odds = 0.25 × 16 = 4.0 after a positive result, and 0.25 × 0.21 = 0.0525 after a negative result. Step 3: probability = odds ÷ (1 + odds), so 4.0 ÷ 5.0 = 0.80 and 0.0525 ÷ 1.0525 = 0.050. The tempting first option skips step 1 and treats the probability of 0.20 as if it were odds (0.20 × 16 = 3.2, then 3.2 ÷ 4.2 = 0.76); a likelihood ratio multiplies odds, so the probability must be converted to odds first and back again at the end. The fourth option reports the LR− itself as a probability. (Section 4, 7.2 From Pre-Test to Post-Test Probability, Equation 5.13 and Worked Example 5.18; Objective 7.2)

6. A regional programme screens adults for a slow-growing cancer. Its annual report says: “Since screening began, twice as many cancers are diagnosed each year, and people whose cancer was found by screening live years longer after diagnosis than people diagnosed after symptoms, which proves that screening works.” Death rates from the cancer in the region have not changed. Which response is best?

Survival from diagnosis is distorted by lead-time bias, length-biased sampling and overdiagnosis. A programme is judged by mortality from the condition in the whole screened population, and here death rates have not changed.
Earlier diagnosis lengthens survival even if death is not delayed, interval screening favours slow-growing cancers, and overdiagnosis adds cancers that would never cause harm. Only mortality in the whole screened population shows real benefit.
The doubling of diagnoses is better explained by overdiagnosis, cancers that would never have caused symptoms or death, than by poor specificity. A PV+ would not show whether screening prevents deaths.
Lead time is a bias when the date of death is unchanged: the person has only known about the cancer for longer. Unchanged death rates suggest the extra survival is not a real gain.
The answer is the second option. Lead-time bias: an earlier diagnosis adds time between diagnosis and death even when death is not delayed. Length-biased sampling: screening at intervals preferentially finds slow-growing cancers, which have a better prognosis. Overdiagnosis: some screen-detected cancers would never have caused symptoms or death, which also helps explain the doubling of diagnoses. Unchanged death rates suggest no real benefit, which is why a programme is evaluated by mortality from the condition in the whole screened population, ideally in a randomized trial. The tempting fourth option counts the extra survival time as a gain, but if the date of death is unchanged, the person has only known about the cancer for longer. (Section 4, 1.2 From a Test to a Screening Program; Objective 1.2)

7. A family practice screens adults for major depression with a questionnaire, and a positive screen leads to a clinical interview. At the chosen cutpoint, Se = 0.88 and Sp = 0.85, so LR+ = 0.88 ÷ 0.15 = 5.87. About 10% of the practice’s adult patients have major depression. What is the probability of depression for a patient who screens positive?

Pre-test odds of 0.10 ÷ 0.90 = 0.111 times LR+ of 5.87 gives 0.652, a probability of 0.652 ÷ 1.652 = 0.39. Most patients who screen positive do not have depression, so the interview confirms it.
This multiplies the probability, not the odds, by the likelihood ratio. Converting 10% to odds of 0.111 first gives post-test odds of 0.652, or a probability of about 0.39.
A screening result with LR+ above 1 does raise the probability of disease. An LR+ of 5.87 multiplies the pre-test odds, taking the probability from 10% to about 39%.
Sensitivity is the probability of a positive screen among people with depression, not the probability of depression among those who screen positive. That depends on the pre-test probability as well.
The answer is the first option. The three steps of Equation 5.13 give: pre-test odds = 0.10 ÷ 0.90 = 0.111; post-test odds = 0.111 × 5.87 = 0.652; post-test probability = 0.652 ÷ (1 + 0.652) = 0.39. A positive screen raises the probability of depression from 10% to about 39%, so about 6 in 10 patients who screen positive do not have major depression, which is why the interview follows. The tempting second option multiplies the probability, not the odds, by the likelihood ratio; the fourth reads the sensitivity as the probability of disease given a positive result, and the third ignores the likelihood ratio. (Section 4, 7.2 From Pre-Test to Post-Test Probability, Equation 5.13 and Worked Example 5.18; Objective 7.2)

✦ Pass the knowledge check with 100% to continue

Section 5

Final Review & Assessment

⏱ Estimated time: 20 minutes

Bringing It All Together

This lesson built up the toolkit for evaluating tests, from the basic distinction between screening (in healthy populations) and diagnosis (in suspected cases), through sensitivity and specificity, into predictive values, and finally into the more sophisticated machinery of cutpoints, ROC curves, and likelihood ratios. The arc moves from how does the test perform? to what does this result mean for this patient in this setting?

The deepest idea in the lesson is that test performance is never just a property of the test. The same sensitivity and specificity produce very different predictive values when prevalence changes, which is why a screening protocol that works in a high-prevalence clinic can collapse into mostly false positives when applied to the general population. Published diagnostic-accuracy studies themselves are subject to design-related bias that inflates reported performance (Lijmer et al., 1999), motivating the QUADAS-2 quality-assessment tool (Whiting et al., 2011) and STARD 2015 reporting standard (Bossuyt et al., 2015). As you finish the assessment, the takeaways below are the practical companions: keep them in mind whenever someone tells you a test is “accurate.”

Key Takeaways from this lesson

  1. You can now tell a screening use of a test, in apparently healthy people whose positive results are sent on for work-up, from a diagnostic use in people already suspected of disease, and you can judge a proposed screening programme against the Wilson and Jungner principles and the criteria of Andermann and colleagues. You can also explain how lead-time bias, length-biased sampling and overdiagnosis make screen-detected cases look as if they do better than they really do. Remember that a programme is judged by mortality from the condition in the whole screened population, not by survival after diagnosis, and that each concern in a scenario (waiting lists, acceptability, cost) points to a named principle.
  2. You can now separate analytic sensitivity, a laboratory detection limit, from diagnostic sensitivity, describe a test’s accuracy (how close the average is to the truth) and precision (how consistent the results are), and calculate and interpret a coefficient of variation, limits of agreement and Cohen’s kappa. For kappa, work step by step (po, then pe, then κ), give its label from Table 5.1, and remember that prevalence and rater bias can pull it down. Above all, keep in mind that close agreement and a small CV show precision only: a test can be consistent and still be consistently wrong.
  3. You can now set out a 2×2 table against a reference standard, calculate sensitivity, specificity and the false negative and false positive fractions, read a confidence interval for them, and explain why they are properties of the test (SnNOut for ruling out, SpPIn for ruling in) that can still shift with the spectrum of disease. You can also explain why the proportion testing positive, the apparent prevalence, differs from the true prevalence, and correct it with the Rogan-Gladen formula. The most useful reminder for the exam is to divide by the disease-status totals (everyone with the disease for Se, everyone without it for Sp), never by the number who tested positive or negative.
  4. You can now calculate PV+ and PV− from a 2×2 table, from the Bayesian formulas and by counting natural frequencies, and explain why they change with prevalence while sensitivity and specificity do not. You can also name the strategies that raise PV+: testing higher-risk groups, using a more specific test and confirming positive results with a second test in series. When a question gives a low prevalence, count out 10,000 people and compare the false positives with the true positives, because that comparison explains a low PV+ more clearly than any formula.
  5. You can now calculate the combined sensitivity and specificity of two tests used in series (positive only if both are positive) and in parallel (positive if either is positive), and choose the strategy that suits the setting. Remember the direction of each change: series lowers Se and raises Sp and PV+, while parallel raises Se and lowers Sp. Then justify your choice by the mistake that matters more, false positives in low-prevalence screening or missed cases where a missed diagnosis would be serious.
  6. You can now explain why moving a cutpoint trades sensitivity against specificity, calculate Youden’s J at several cutpoints and find the one it favours, read a ROC curve, and interpret its AUC as the probability that a randomly chosen diseased person has a higher test value than a randomly chosen healthy person. Remember that J weights Se and Sp equally, so when a missed case is more costly you move to a lower cutpoint, and when a false positive is more costly you move to a higher one.
  7. You can now calculate LR+, LR− and category-specific likelihood ratios, interpret each in a sentence, and use them to move from a pre-test probability to a post-test probability for one person. The most important reminder is to convert the probability to odds before multiplying by the LR and to convert the odds back to a probability afterwards, and then to say whether the result rules disease in or out (an LR+ above 10, or an LR− below 0.1, is strong evidence).

Final Knowledge Assessment

Complete all 17 questions below with 100% accuracy to finish this lesson. The final assessment can be submitted once every section knowledge check has been passed.

Final Assessment: Screening and Diagnostic Tests

1. A province offers a point-of-care syphilis test to every pregnant person at the first prenatal visit, and anyone who tests positive then has laboratory serology. At a sexual health clinic, a physician orders laboratory serology for a man with a painless genital ulcer. How should each use of testing be classified?

A positive result leading to treatment does not make a use diagnostic. The prenatal test is offered to all pregnant people regardless of symptoms to find infection early, which is screening.
This uses being a patient in care, rather than suspicion of disease, as the criterion. The prenatal test is offered to everyone without regard to symptoms (screening); the man with an ulcer is suspected of syphilis (diagnosis).
Screening applies a test to apparently healthy people to detect disease early, with positives sent on for work-up. Diagnosis confirms or rules out disease in a person already suspected, like the man with an ulcer.
The classification is right, but positive screens are followed by diagnostic work-up, here laboratory serology. A screening test is not meant to be the final word.
The answer is the third option. A screening test is applied to apparently healthy people without symptoms to detect disease early, and those who test positive go on to diagnostic work-up; a diagnostic test is applied to people already suspected of disease, to confirm or rule it out. Pregnant people are tested whether or not they have symptoms, so the prenatal test is screening; the man with an ulcer is tested because syphilis is suspected, so his test is diagnostic. The tempting second option uses “being a patient” rather than suspicion of disease as the criterion, and the fourth is wrong because positive screens are followed by diagnostic testing. (Section 1, 1.1 Screening Tests and Diagnostic Tests, cards “Screening Tests” and “Diagnostic Tests”; Objective 1.1)

2. A control serum is tested 10 times on a laboratory analyser, giving a mean of 4.00 mmol/L and a standard deviation of 0.12 mmol/L. A technologist says: “The coefficient of variation is small, so the analyser gives the right answer.” Which response is correct?

CV = σ ÷ μ = 0.12 ÷ 4.00 = 0.03, a measure of precision. Accuracy is a separate property, checked by comparing the mean of repeated results with the serum's known value.
A small CV shows that results are consistent, which is precision. An analyser can be precise but inaccurate, the tight off-centre cluster of Figure 5.1, so accuracy needs comparison with the known value.
This divides the wrong way round. The coefficient of variation is the standard deviation divided by the mean, 0.12 ÷ 4.00 = 0.03, a small value showing good precision.
The standard deviation is not the coefficient of variation; CV divides it by the mean, giving 0.03. A small CV points to good precision, and accuracy cannot be judged from either value alone.
The answer is the first option. CV = σ ÷ μ = 0.12 ÷ 4.00 = 0.03, or 3%, a small value that shows the repeated results are consistent, which is precision. Accuracy is a different property, whether the mean of repeated results is close to the true value, and an analyser can be precise but inaccurate (the tight, off-centre cluster of Figure 5.1); to check accuracy, the mean of 4.00 mmol/L has to be compared with the serum’s known concentration. The tempting second option treats a small CV as evidence of accuracy, which does not follow; the third option divides the wrong way round. (Section 1, 2.2 Accuracy and Precision and 2.3 Measuring Precision When a Test Reports a Number, tab “Coefficient of variation”, Worked Example 5.1; Objectives 2.2 and 2.3)

3. In clinics without a reading device, the point-of-care syphilis strip is read by eye. Two technologists independently read the same 150 strips, with the results below. Calculate Cohen’s kappa and choose the correct conclusion.

Technologist B: positiveTechnologist B: negativeTotal
Technologist A: positive24630
Technologist A: negative6114120
Total30120150
0.92 is the observed agreement, not kappa, and agreement between readers is not accuracy. Two readers who make the same mistakes agree closely and are both wrong.
κ = (0.92 − 0.68) ÷ (1 − 0.68) = 0.75, substantial in Table 5.1. Kappa compares the readers with each other, so it measures precision; accuracy needs a reference standard.
0.75 falls in the 0.61 to 0.80 band, which Table 5.1 labels substantial, not almost perfect. Agreement also cannot replace a reference test, because consistent readers can share the same errors.
0.24 is only the numerator, po − pe. Kappa divides it by 1 − pe = 0.32, which gives 0.75.
The answer is the second option. Observed agreement po = (24 + 114) ÷ 150 = 0.92. Each technologist calls 30 of the 150 strips positive (0.20), so chance agreement pe = (0.20 × 0.20) + (0.80 × 0.80) = 0.04 + 0.64 = 0.68, and κ = (0.92 − 0.68) ÷ (1 − 0.68) = 0.24 ÷ 0.32 = 0.75, in the 0.61 to 0.80 band of Table 5.1 (substantial). Kappa compares the two technologists with each other, not with the truth: two readers who make the same mistakes agree closely and are both wrong, so accuracy must be judged against a reference standard, through sensitivity and specificity. The tempting first option reports po as kappa and treats agreement as accuracy; the fourth stops after the numerator, po − pe. (Section 1, 2.4.1 Agreement Beyond Chance, Equation 5.3, Table 5.1; Objective 2.4)

4. Two physiotherapists independently rate the gait of the same 100 older adults as normal, mildly impaired or severely impaired. Cohen’s (unweighted) kappa is 0.58, and with linear weights the weighted kappa is 0.69. Most of their disagreements are between neighbouring categories. Which explanation is correct?

Both unweighted and weighted kappa remove the agreement expected by chance. They differ only in how disagreements are weighted.
Correcting for prevalence is what PABAK does, not weighted kappa. Weighted kappa gives partial credit to disagreements according to how far apart the categories are.
Weighted kappa counts near disagreements as partial agreement, so it rises when most disagreements are one category apart. It suits ordered ratings, and since the weights change the value, they should be reported.
The two values are expected to differ for ordered categories. Unweighted kappa treats every disagreement as equally serious, while weighted kappa gives partial credit to neighbouring ones.
The answer is the third option. The unweighted kappa treats every disagreement as equally serious, whereas a weighted kappa gives each cell a disagreement weight that grows with the distance between the two categories chosen (linear weights are |i − j| ÷ (k − 1)), so neighbouring disagreements count as partial agreement. When most disagreements are one category apart, as here, the weighted kappa is higher. It suits ordinal ratings, and because the choice of weights changes the value, a report states which weights were used. The tempting first option is wrong because both versions remove the agreement expected by chance; they differ only in how disagreements are weighted. (Section 1, 2.4.3 Weighted Kappa; Objective 2.4)

5. The point-of-care syphilis test is evaluated against laboratory serology (the reference standard) in 500 clients of a sexual health clinic. The columns give the reference result and the rows the point-of-care result. Which set of values is correct?

Syphilis (laboratory)No syphilis (laboratory)Total
Point-of-care positive8816104
Point-of-care negative12384396
Total100400500
This swaps the two directions: 88 ÷ 104 and 384 ÷ 396 divide by the row totals, which gives the predictive values. Sensitivity and specificity divide by the reference columns, 88 ÷ 100 and 384 ÷ 400.
Se and Sp are right, but predictive values are not equal to them. PV+ divides the true positives by all positive results (88 ÷ 104 = 0.85), and PV− the true negatives by all negative results (384 ÷ 396 = 0.97).
This swaps sensitivity with specificity and PV+ with PV−. Se is the share of people with syphilis who test positive, 88 ÷ 100 = 0.88, and PV+ the share of positives who have syphilis, 88 ÷ 104 = 0.85.
Se and Sp are read down the reference columns (88 ÷ 100, 384 ÷ 400), and the predictive values across the test rows (88 ÷ 104, 384 ÷ 396). Each denominator names the group the measure describes.
The answer is Se = 0.88, Sp = 0.96, PV+ = 0.85 and PV− = 0.97. Sensitivity and specificity are read down the reference columns: Se = 88 ÷ 100 = 0.88 and Sp = 384 ÷ 400 = 0.96. Predictive values are read across the test rows: PV+ = 88 ÷ 104 = 0.85 and PV− = 384 ÷ 396 = 0.97. The tempting first option swaps the two directions, dividing by the row totals for Se and Sp and by the column totals for the predictive values; asking which group each denominator describes (everyone with syphilis, or everyone who tested positive) avoids the error. The second option assumes that predictive values equal Se and Sp. (Section 2, 3.2.2 Sensitivity, Specificity and the Error Fractions, and Section 3, 4.2.1 From the 2×2 Table and the Bayesian Formulas; Objectives 3.2 and 4.2)

6. In the clinic study (prevalence of syphilis 20%), the point-of-care test had Se = 0.88, Sp = 0.96 and PV+ = 0.85. A colleague proposes to quote all three values as properties of the test in guidance for prenatal clinics, where far fewer of the people tested have syphilis. Which response is correct?

Using the same reference standard does not make every measure a property of the test. PV+ is calculated across the test-positive row, which mixes people with and without syphilis in proportion to the prevalence.
Se and Sp are calculated within the disease-status columns, so prevalence does not enter them. PV+ mixes the two groups in proportion to prevalence, so with far fewer infected people it falls.
Prevalence does not enter sensitivity or specificity, because each is calculated within one disease-status group. Only the predictive values, read across the rows, depend on prevalence.
This has the direction backwards. Fewer infected people means fewer true positives, while the false positives come from the uninfected group, which is larger, so PV+ falls.
The answer is the second option. Sensitivity and specificity are computed down the disease-status columns, within the people who do and do not have syphilis, so the clinic’s 20% prevalence does not enter them; predictive values are computed across the test-result rows, which mix the two groups in proportion to the prevalence. Where prevalence is much lower, the uninfected group is larger, so false positives grow relative to true positives and PV+ falls. The tempting fourth option has the direction backwards: fewer infected people means fewer true positives, while the false positives come from the large uninfected group. (Section 2, 3.3 Sensitivity and Specificity as Properties of the Test and Figure 5.2; Section 3, 4.3 How Prevalence Changes Predictive Values; Objectives 3.3 and 4.3)

7. In prenatal clinics, where about 0.5% of pregnant people have syphilis, the point-of-care test (Se = 0.88, Sp = 0.96) is positive in 4.4% of those tested. A provincial dashboard reports the 4.4% as “the prevalence of syphilis in pregnancy”. Which statement is correct?

Dividing by sensitivity corrects only for missed cases. When a condition is rare, false positives among the large uninfected group dominate, so the 4.4% overstates the true prevalence of about 0.5%.
A specificity of 0.96 still means 4% of the 99.5% uninfected test positive, about 3.98 percentage points. Almost all of the 4.4% comes from false positives, so it is not a reasonable estimate of prevalence.
The direction is right but the cause is wrong. A sensitivity below 1 misses cases and pushes the apparent prevalence down; the overstatement comes from the false positives, which depend on specificity.
AP = (0.005 × 0.88) + (0.995 × 0.04) = 0.044, almost all of it false positives. The figure is a positivity rate; Rogan-Gladen gives 0.004 ÷ 0.84 = 0.005, the true prevalence.
The answer is the fourth option. The apparent prevalence is AP = P × Se + (1 − P)(1 − Sp) = (0.005 × 0.88) + (0.995 × 0.04) = 0.0044 + 0.0398 = 0.044, so almost all of the 4.4% comes from false positives among the 99.5% who are uninfected, and the true positives contribute only 0.44 percentage points. The Rogan-Gladen formula recovers the true prevalence: (0.044 + 0.96 − 1) ÷ (0.88 + 0.96 − 1) = 0.004 ÷ 0.84 = 0.005, about 0.5%. The tempting first option corrects only for the missed cases, but when a condition is rare the false positives, which depend on the specificity, dominate; the third option blames the sensitivity, which pushes the figure down, not up. (Section 2, 3.4 True and Apparent Prevalence and 3.5 Estimating True Prevalence: The Rogan-Gladen Formula, Equations 5.6 and 5.7; Objectives 3.4 and 3.5)

8. For 10,000 pregnant people screened with the point-of-care syphilis test (Se = 0.88, Sp = 0.96) at a prevalence of 0.5%, what are the expected numbers of true positives and false positives, and the PV+?

Of 10,000, 50 have syphilis and 0.88 × 50 = 44 test positive; 4% of the 9,950 without it gives 398 false positives. PV+ = 44 ÷ 442 = 0.10, because the uninfected group is so large.
This uses the 6 missed infections as the false positives. False positives come from the 9,950 people without syphilis, and 4% of them is 398, so PV+ is far lower than 0.88.
This forgets that the test misses some infections: only 88% of the 50 people with syphilis test positive, giving 44 true positives, not 50.
The counts are right, but PV+ = 44 ÷ 442 = 0.10, not 0.85. The clinic's PV+ belongs to its 20% prevalence and does not carry over to a prevalence of 0.5%.
The answer is 44 true positives, 398 false positives and PV+ = 0.10. Of 10,000 people, 0.5% (50) have syphilis, and the test detects 88% of them: 0.88 × 50 = 44 true positives (6 are missed). The other 9,950 do not have syphilis, and 4% of them (1 − Sp) test positive: 0.04 × 9,950 = 398 false positives. PV+ = 44 ÷ (44 + 398) = 44 ÷ 442 = 0.10, so about nine in ten positive screens are false alarms, because the uninfected group is so large; the PV+ of 0.85 in the clinic does not carry over. The tempting second option uses the 6 missed infections as if they were the false positives, but false positives come from the people without syphilis; the third option forgets that the test misses some infections. (Section 3, 4.2.2 Counting Natural Frequencies, Worked Example 5.10 and Figure 5.3; Objectives 4.2 and 4.3)

9. Laboratory serology has Se = 0.95 and Sp = 0.99 for syphilis, and the point-of-care test has Se = 0.88 and Sp = 0.96; assume that their errors are independent. In series, Se = Se1 × Se2 and Sp = 1 − (1 − Sp1)(1 − Sp2); in parallel, Se = 1 − (1 − Se1)(1 − Se2) and Sp = Sp1 × Sp2. Which statement is correct?

This swaps the two strategies. In series both tests must be positive, so sensitivity falls to 0.88 × 0.95 = 0.84 and specificity rises to 0.9996; parallel testing does the reverse.
This mixes the formulas. Parallel testing raises sensitivity to 0.99 but lowers specificity to 0.96 × 0.99 = 0.95; no combination improves both measures, and series specificity is 0.9996.
In series a person without syphilis is called positive only if both tests err, so Sp rises to 0.9996 and PV+ with it. The price is a sensitivity of 0.88 × 0.95 = 0.84.
The numbers are right but the conclusion is not. Parallel testing lowers specificity, adding false positives, so it lowers PV+; it suits settings where a missed case is the costly error.
The answer is the third option. Series: Se = 0.88 × 0.95 = 0.836 and Sp = 1 − (0.04 × 0.01) = 1 − 0.0004 = 0.9996. Parallel: Se = 1 − (0.12 × 0.05) = 1 − 0.006 = 0.994 and Sp = 0.96 × 0.99 = 0.950. In series a person without syphilis is called positive only if both tests err, so specificity and PV+ rise, which matters most in a low-prevalence screen; the price is a lower sensitivity (about 16% of infections missed). The tempting fourth option has the right numbers but the wrong conclusion: parallel testing lowers specificity, adds false positives and so lowers PV+; it suits a setting where a missed case is the costly error. (Section 3, 5.1 Combining Two Tests: Series and Parallel, Worked Example 5.12; Objective 5.1)

10. A blood marker for liver fibrosis is evaluated at four cutpoints, with a score at or above the cutpoint called positive. A positive result leads to a liver biopsy, which carries a risk of serious bleeding, while the disease progresses slowly and a missed case can be found at the next yearly test. Which answer is correct?

Cutpoint (score ≥)1.01.52.02.5
Sensitivity0.960.890.780.60
Specificity0.550.720.860.95
Youden's J is Se + Sp − 1, not the highest sensitivity; at 1.0 it is 0.96 + 0.55 − 1 = 0.51, the smallest of the four. J is largest at 2.0.
J is right, but lowering the cutpoint suits a setting where misses are the costly error. Here a false positive means a risky biopsy and a missed case can be found next year, so specificity deserves more weight.
0.95 is the specificity at 2.5, not J. There J = 0.60 + 0.95 − 1 = 0.55, smaller than the 0.64 at 2.0.
J = 0.78 + 0.86 − 1 = 0.64 at 2.0, but J weights Se and Sp equally. When false positives are the costly error, a higher cutpoint raises specificity (0.95 at 2.5) at an acceptable cost in missed cases.
The answer is the fourth option. Youden’s J = Se + Sp − 1: 0.96 + 0.55 − 1 = 0.51; 0.89 + 0.72 − 1 = 0.61; 0.78 + 0.86 − 1 = 0.64; 0.60 + 0.95 − 1 = 0.55, so J favours a cutpoint of 2.0. J weights sensitivity and specificity equally; when a false positive is the more costly error, as with a risky biopsy, specificity should get more weight, which means a higher cutpoint (Sp = 0.95 at 2.5), at the price of missing more cases, acceptable here because a missed case can be found at the next test. The tempting second option moves the cutpoint in the direction that suits costly misses, not costly false positives, and the third confuses J with the specificity. (Section 4, 6.3 Choosing a Cutpoint with Youden’s J, Worked Example 5.14 and the panel “Choosing the Optimal Cutpoint”; Objective 6.3)

11. A study reports that a new urine test for bladder cancer has an area under the ROC curve (AUC) of 0.76. A colleague writes: “The test gives the correct result for 76% of patients.” Which interpretation is correct?

The proportion classified correctly depends on the cutpoint chosen and on prevalence. The AUC depends on neither: it summarises discrimination across all cutpoints.
The share of positive results that are true is PV+, which depends on prevalence. The AUC is the probability that a random person with the disease has a higher test value than a random person without it.
The AUC is the probability that a randomly chosen patient with the disease has a higher value than one without it. 0.76 lies in the 0.70 to 0.80 band of Table 5.4, acceptable discrimination.
The AUC is not a sensitivity at any particular cutpoint; it summarises the test across all of them. A value of 0.76 is acceptable, not excellent, in Table 5.4.
The answer is the third option. The AUC summarises the test across all cutpoints: it is the probability that a randomly selected person with the disease has a higher test value than a randomly selected person without it, so 0.5 means chance and 1.0 means perfect separation. A value of 0.76 lies in the 0.70 to 0.80 band of Table 5.4, acceptable discrimination. The colleague’s reading is tempting but wrong, because the proportion of patients classified correctly depends on the cutpoint chosen and on the prevalence, whereas the AUC depends on neither; the second option describes PV+. (Section 4, 6.2 ROC Curves and the Area Under the Curve, Table 5.4; Objective 6.2)

12. A man with a genital ulcer and a recent partner with syphilis has a pre-test probability of syphilis of 40%. The point-of-care test has Se = 0.88 and Sp = 0.96, so LR+ = 0.88 ÷ 0.04 = 22 and LR− = 0.12 ÷ 0.96 = 0.125. What is his probability of syphilis after a positive result and after a negative result?

These are the clinic study's predictive values, which belong to its 20% prevalence. For a man with a pre-test probability of 40%, the likelihood ratios must be applied to his own pre-test odds.
Pre-test odds are 0.40 ÷ 0.60 = 0.667. Multiplying by 22 gives 14.7 (94%), and by 0.125 gives 0.083 (7.7%), converting back with odds ÷ (1 + odds).
This multiplies the probability instead of the odds. A likelihood ratio acts on odds, so 40% is first converted to 0.667, and the result never reaches 100%.
The positive result is right, but 12% is the false negative fraction, the share of people with syphilis who test negative. After a negative result his odds become 0.667 × 0.125 = 0.083, or 7.7%.
The answer is 94% and 7.7%. Step 1: pre-test odds = 0.40 ÷ 0.60 = 0.667. Step 2: post-test odds = 0.667 × 22 = 14.7 after a positive result, and 0.667 × 0.125 = 0.083 after a negative result. Step 3: probability = odds ÷ (1 + odds), so 14.7 ÷ 15.7 = 0.94 and 0.083 ÷ 1.083 = 0.077. The tempting first option borrows the predictive values of the clinic study, but those belong to the clinic’s 20% prevalence; for a man whose pre-test probability is 40%, the likelihood ratios must be applied to his own pre-test odds. The third option multiplies the probability instead of the odds. (Section 4, 7.2 From Pre-Test to Post-Test Probability, Equation 5.13 and Worked Example 5.18; Objective 7.2)

13. For the same man (LR+ = 22, LR− = 0.125, probability of syphilis 7.7% after a negative result), is the point-of-care test better at ruling syphilis in or ruling it out, and should a negative result rule it out? As a rough guide, an LR+ above 10 strongly rules disease in and an LR− below 0.1 strongly rules it out.

A sensitivity of 0.88 is not high enough for SnNOut in a high-risk man: the LR− of 0.125 is above the guide of 0.1, and a negative result still leaves about 8% probability.
Every LR− below 1 lowers the odds, but that alone does not make a test good at ruling out. The LR− of 0.125 is above the guide of 0.1, while the LR+ of 22 is well above 10.
Ruling in is right, but whether a negative rules out depends on the remaining probability and the stakes. About 8 in 100 such men would still have an infection that can be passed on and is easily treated.
The LR+ of 22 strongly raises the odds, matching the higher specificity (SpPIn), while the LR− of 0.125 lowers them less decisively. At about 8% after a negative result, serology should follow.
The answer is the fourth option. The LR+ of 22 is well above the guide of 10, so a positive result strongly raises the odds of syphilis, whereas the LR− of 0.125 is above the guide of 0.1, so a negative result lowers the odds less decisively; this matches the specificity (0.96) being higher than the sensitivity (0.88), as in SpPIn. After a negative result, about 8 in 100 men like him would still have syphilis, too many to stop investigating an infection that can be passed on and is easily treated. The tempting first option applies SnNOut to a sensitivity that is not high enough to rule disease out in a person with a high pre-test probability, and the second forgets that every LR− below 1 lowers the odds without necessarily ruling disease out. (Section 4, 7.1 Likelihood Ratios and 7.2 From Pre-Test to Post-Test Probability; Section 2, 3.2.2 Sensitivity, Specificity and the Error Fractions (cards) and 3.3 Sensitivity and Specificity as Properties of the Test; Objectives 7.1 and 3.3)

14. C-reactive protein (CRP) is measured in 400 young children with fever, 80 of whom have a serious bacterial infection on the reference work-up. A child with fever whose pre-test probability of serious infection is 10% has a high CRP. What is the likelihood ratio for a high result, and the child’s post-test probability?

CRP resultSerious infection (n = 80)No serious infection (n = 320)
High (80 mg/L or more)4816
Intermediate (20 to 79 mg/L)2496
Low (below 20 mg/L)8208
This divides the raw counts, ignoring that four times as many children were without serious infection. Each count must first become a proportion of its own group: (48 ÷ 80) ÷ (16 ÷ 320) = 12.
The category-specific LR is (48 ÷ 80) ÷ (16 ÷ 320) = 0.60 ÷ 0.05 = 12. Pre-test odds of 0.111 × 12 = 1.33 convert to a probability of 1.33 ÷ 2.33 = 0.57.
The LR is right, but it multiplies the odds, not the probability. Odds of 0.111 × 12 = 1.33 convert back to 57%; a probability can never exceed 1.
0.60 is only the numerator, the proportion with serious infection who have a high CRP. The LR divides it by the proportion without serious infection who have a high CRP (0.05), giving 12.
The answer is LR = 12 and 57%. A category-specific likelihood ratio divides the proportion of children with serious infection who have that result by the proportion of children without serious infection who have the same result: (48 ÷ 80) ÷ (16 ÷ 320) = 0.60 ÷ 0.05 = 12. Then pre-test odds = 0.10 ÷ 0.90 = 0.111, post-test odds = 0.111 × 12 = 1.33, and post-test probability = 1.33 ÷ 2.33 = 0.57. The tempting first option divides the raw counts (48 ÷ 16), which ignores that four times as many children were without serious infection; each count must first be turned into a proportion of its own group. (An intermediate CRP has an LR of (24 ÷ 80) ÷ (96 ÷ 320) = 1.0 and leaves the probability unchanged.) (Section 4, 7.1 Likelihood Ratios, tab “Category-Specific LR”, and 7.2 From Pre-Test to Post-Test Probability, Equation 5.12, Worked Examples 5.17 and 5.18; Objectives 7.1 and 7.2)

15. A ministry proposes screening adults aged 40 to 70 for thyroid cancer with a neck ultrasound, followed by a needle biopsy for anyone with a suspicious nodule. Planners learn that the waiting list for biopsies is already eight months long, and that specialists disagree about whether small, slow-growing cancers found this way should be treated at all. Which Wilson and Jungner principles do these two concerns illustrate?

The waiting list means facilities for diagnosis and treatment are not available, and disagreement over treating small, slow-growing cancers means there is no agreed policy on whom to treat, a warning of overdiagnosis.
These principles concern the test itself, but neither concern is about the ultrasound. One is about capacity for follow-up biopsies and the other about whether to treat what is found.
Neither concern is about whether thyroid cancer is an important health problem or about case-finding being continuous. They concern follow-up facilities and the policy on whom to treat.
An accurate test is only one requirement. The Wilson and Jungner principles also ask about the condition, its treatment and the programme, including facilities and an agreed treatment policy.
The answer is the first option. The eight-month waiting list for the follow-up biopsy means that facilities for diagnosis and treatment are not available for the people a screening programme would refer (principle 3), and the specialists’ disagreement about treating small, slow-growing cancers means that there is no agreed policy on whom to treat as patients (principle 8); it also warns of overdiagnosis. The tempting second option names the principles about the test itself, but neither concern is about the ultrasound. The fourth option is wrong because an accurate test is only one requirement: the principles also ask about the condition, its treatment and the programme. (Section 4, 1.2.1 Principles for Screening and 1.2.2 Applying the Criteria: Colorectal Cancer Screening, Worked Example 5.13; Objective 1.2)

16. A province screens every newborn for permanent hearing loss, which affects about 2 in 1,000 newborns. For 10,000 newborns, the first test alone (otoacoustic emissions) gives about 19 true positives and 998 false positives. Retesting the babies who fail it with a second test, and referring only those who fail both (series testing), gives about 18 true positives and 30 false positives. Which statement is correct?

This confuses PV+ with sensitivity. PV+ divides the true positives by everyone who fails: 19 ÷ 1,017, about 0.02, so most babies who fail the first test hear normally.
PV+ = 19 ÷ 1,017, about 0.02, for the first test and 18 ÷ 48, about 0.38, in series. Requiring both tests to be failed raises specificity and removes most false positives, with sensitivity falling slightly.
This has series testing backwards. Requiring both tests to be failed lowers sensitivity (19 to 18 true positives) and raises specificity; it is parallel testing that raises sensitivity.
The referrals removed by series testing are mostly false positives (998 down to 30), so the share of referred babies who truly have hearing loss rises to about 0.38.
The answer is the second option. PV+ is the true positives divided by everyone who tests positive: 19 ÷ (19 + 998) = 0.019 for the first test alone and 18 ÷ (18 + 30) = 0.38 in series. Because hearing loss is rare, the first test alone refers about 98 hearing babies for every 2 with hearing loss; requiring both tests to be failed raises specificity and cuts the false positives sharply, while sensitivity falls slightly (19 to 18 true positives). Most referred babies still have normal hearing, so families should be told that a referral is not a diagnosis. The tempting third option has the direction of series testing backwards: it lowers sensitivity and raises specificity. The first option confuses PV+ with sensitivity, and the fourth ignores that the referrals removed are mostly false positives. (Section 3, 4.2.2 Counting Natural Frequencies and 5.1 Combining Two Tests: Series and Parallel, Worked Example 5.12; Objectives 4.2 and 5.1)

17. For the same newborn hearing programme, a colleague proposes giving every baby both tests and referring any baby who fails either one (parallel testing), so that fewer babies with hearing loss are missed. For 10,000 newborns this gives about 20 true positives and 1,267 false positives, against about 18 true positives and 30 false positives in series. Which assessment is correct?

This looks only at sensitivity. Two more babies found per 10,000 cost more than 1,200 extra referrals of babies with normal hearing, which strain diagnostic facilities and worry families.
The strategies differ sharply: parallel testing finds about 20 cases with 1,267 false positives, while series finds about 18 with 30. The choice is about which mistake matters more.
Parallel testing lowers specificity, adding false positives, so it lowers PV+: about 20 ÷ 1,287 = 0.016, against 0.38 in series.
Parallel testing raises sensitivity but lowers specificity. In low-prevalence screening false positives dominate, and the extra referrals would overwhelm facilities for diagnosis, so series testing fits better.
The answer is the fourth option. Parallel testing raises sensitivity but lowers specificity, so it finds slightly more cases (about 20 against 18) at the price of far more false positives (about 1,267 against 30), and its PV+ falls to about 20 ÷ 1,287 = 0.016. The extra referrals would strain the facilities for diagnosis (a Wilson and Jungner principle) and cause needless worry. Choosing between series and parallel testing is a choice about which mistake matters more: in low-prevalence screening the false positives dominate, while parallel testing suits a setting where a missed case would be very serious and extra positives can be resolved easily. The tempting first option looks only at sensitivity; the third is wrong because parallel testing adds false positives and so lowers PV+. (Section 3, 5.1 Combining Two Tests: Series and Parallel; Section 4, 1.2.1 Principles for Screening; Objectives 5.1 and 1.2)

✦ Pass every section knowledge check before submitting