HSCI 410 · Lesson 2

Data Cleaning &
Descriptive Analyses

Exploratory Data Analysis For Epidemiology

Learning objectives for this lesson:

  1. Identify common types of data errors, including coding faults in a raw data file, the stage of the data pipeline where each enters, and how each should be corrected.
  2. Classify a described pattern of missing data as missing completely at random (MCAR), missing at random (MAR) or missing not at random (MNAR), justify the classification, and explain the consequence of a complete-case analysis under each.
  3. Apply systematic cleaning checks (range, consistency and outlier checks), decide whether to keep, transform or exclude each flagged value, and record the rule in a cleaning log.
  4. Choose a method for handling missing data (complete-case analysis, single imputation or multiple imputation) that suits the plausible mechanism and the amount missing.
  5. Choose, obtain in R and interpret measures of central tendency, spread and shape (skewness and excess kurtosis) for each variable type and distribution, and describe the shape of the distribution that a summary implies.
  6. Construct and interpret descriptive tables for epidemiologic research: frequency tables, cross-tabulations with an odds ratio, and a Table 1 stratified by exposure.
  7. Select visualisations that suit the variable type and the question, and assess whether a variable is close enough to normal for parametric methods.

This course was developed by Dr. Kiffer G. Card based on Dohoo, I. R., Martin, S. W., & Stryhn, H. (2012). Methods in Epidemiologic Research. VER Inc.

Reference

Glossary: Key Terms, People & Concepts

📚 Reference page, available throughout the lesson

This glossary collects the key concepts, people, and ideas you will meet in this lesson. Use it as a reference while you work through the material, or as a review before assessments. Type in the search box to filter entries.

Key Concepts & Ideas
Tidy Data A data layout where each variable is a column, each observation is a row, and each cell holds a single value. Coined by Hadley Wickham; the foundation of the tidyverse workflow.
Data Pipeline The full sequence of steps that takes raw data through cleaning, transformation, and analysis to a final result. Errors can enter at any stage; tracking them is essential.
Codebook / Data Dictionary Documentation that describes every variable in a dataset (name, type, allowable values, units, coding, source). Required for reproducible analysis.
Outlier An observation that lies far from the rest of the distribution. May reflect a genuine extreme value, a measurement error, or a data-entry mistake; investigate before removing.
Range Check Verifying that values for a variable fall within plausible biological or logical limits (e.g., human age between 0 and 120). A first-pass tool for spotting impossible entries.
Frequency Table A tabular summary showing the count (and often the percentage) of each level of a categorical variable. The simplest descriptive tool for nominal/ordinal data.
Missingness Patterns of missing values in a dataset. Classified by mechanism (MCAR/MAR/MNAR); the mechanism governs whether complete-case analysis or imputation is appropriate.
Type Conversion (Coercion) Changing a variable's data type (e.g., character to numeric, factor to integer). Mistyped variables are a common source of analysis errors in R.
Derived Variable A new variable computed from existing ones (e.g., BMI from height and weight, age categories from age). Document the formula in the data dictionary.
Methods & Statistical Concepts
Mean The arithmetic average. Sensitive to outliers; appropriate for symmetric continuous data.
Median The 50th percentile, the value that splits the distribution in half. Robust to outliers and skew; preferred for non-normal continuous data.
Standard Deviation (SD) A measure of spread equal to the square root of the variance. Reported alongside the mean for symmetric data.
Interquartile Range (IQR) The difference between the 75th and 25th percentiles (Q3 − Q1). A robust measure of spread; reported with the median.
Q-Q Plot A quantile-quantile plot comparing observed data to a theoretical distribution (often the normal). Points falling on the diagonal suggest the distribution fits.
Histogram A bar chart of binned counts that displays the empirical distribution of a continuous variable. Useful for spotting skew, modality, and outliers.
Box Plot A graphical summary showing the median, the IQR (box), whiskers that reach the most extreme values within 1.5×IQR of the box, and points beyond the whiskers. Compact for comparing groups.
Skewness Asymmetry of a distribution. Positive (right) skew has a long right tail; negative (left) skew has a long left tail. Influences choice of mean vs. median.
Multiple Imputation A principled method for handling missing data: generate multiple plausible values per missing cell, analyse each completed dataset, and pool results using Rubin's rules.
Odds and Odds Ratio Odds compare the number of people with a characteristic with the number without it. An odds ratio divides the odds in one group by the odds in another; a value of 1 means no association.
Bias A systematic difference between an estimate and the true value, so that repeated studies would be wrong in the same direction on average.
Precision and Standard Error Precision describes how much an estimate would vary from sample to sample. The standard error measures that variation; for a mean it is the SD divided by the square root of the sample size.
Confidence Interval A range around an estimate, roughly the estimate plus or minus two standard errors for a 95% interval, that shows the values compatible with the data.
Null Hypothesis and p-Value The null hypothesis is the starting assumption of no difference or no effect. The p-value is the probability of a result at least as extreme as the one observed if the null hypothesis were true.
Standardised Difference The difference between two groups divided by a pooled standard deviation, used in Table 1 to describe the size of an imbalance. Values below about 0.1 are usually read as negligible.
Confounder A variable related to both the exposure and the outcome that can distort the apparent association between them.
Complete-Case Analysis Analysing only the rows with no missing value on any variable used; also called listwise deletion. Defensible when little is missing and MCAR is plausible.
Parametric and Rank-Based Methods Parametric methods assume the data follow a particular distribution, usually the normal. Rank-based (non-parametric) methods work with the ranks of the values and make fewer assumptions.
Geometric Mean The back-transformed mean of logged values, exp(mean(log(x))); a typical value for right-skewed data such as biomarker concentrations.
Regular Expression A short text pattern, such as ^vanc for text that starts with vanc, used to find or replace variants in text fields.
Eigenvalue In factor analysis, the amount of shared variation among the items that one factor accounts for. Parallel analysis keeps factors whose eigenvalues exceed those from random data.
Factor Loading The correlation between an item and a factor. This lesson treats loadings of 0.40 or higher as meaningful.
psych (R package) An R package for survey and psychological measurement. This lesson uses its describe() function for detailed descriptive statistics and its alpha() and fa.parallel() functions for scale checks.
alpha() The psych function that calculates Cronbach's alpha for a set of items and shows what alpha would be if each item were dropped.
fa.parallel() The psych function for parallel analysis, which compares the eigenvalues of the real items with those of random data to suggest how many factors to keep.
factanal() The base R function for factor analysis. It reports each item's loading on each factor and needs rows without missing values.
mice (R package) An R package for multiple imputation by chained equations: mice() creates completed datasets, with() analyses each one, and pool() combines the results with Rubin's rules.
Key People
John Tukey (1915–2000) American statistician who coined the term “exploratory data analysis” (EDA) and invented the box plot, stem-and-leaf display, and many tools for visual data inspection.
Hadley Wickham (1979– ) New Zealand statistician and chief scientist at Posit (RStudio). Created ggplot2, dplyr, tidyr, and the broader tidyverse; defined “tidy data”.
No matching entries. Try a different search term.
Section 1

Data Quality & Types of Errors

⏱ Estimated time: 35 minutes
Lesson 2 · HSCI 410

Data Cleaning & Descriptive Analyses

Every regression coefficient in later lessons rests on data someone had to clean first.

The R activities

Code blocks: logic before syntax

Three R code blocks run through the full pipeline on a simulated survey dataset called phaa_survey.csv.

  • Block 1: read the raw file and compare it with the codebook.
  • Block 2: fix variable types and apply range checks.
  • Block 3: calculate descriptives, produce standard visualisations, assess internal consistency and build derived scale scores.

The narration describes the logic. The written lesson has the syntax, a primer on reading R code, and annotated output.

Section 1 of 3

Data Quality & Types of Errors

The error taxonomy, the data pipeline, and the three missing-data mechanisms that govern your analytic choices.

The pipeline

Where errors enter

Collection Entry Storage Analysis errors can enter at every stage

Data quality is multidimensional: accuracy, completeness, timeliness, and consistency can all break down at any point (Wang & Strong, 1996).

Error types

Five things that go wrong

Transcription

Digits transposed, decimal-point shifts during copying.

Coding

Incorrect code assignment; category labels reversed.

Logical inconsistency

Cross-field contradictions (e.g., diagnosis before birth).

Out-of-range

Values outside biological or logical plausibility limits.

Duplicates

The same observation entered more than once.

Van den Broeck et al. (2005): cleaning = screening → diagnosing → editing, planned into the study design.

Missing data

Three mechanisms (Rubin, 1976)

MCAR

Missingness unrelated to any data value, observed or not. Complete-case analysis is unbiased (but less precise).

MAR

Missingness depends on observed variables only. Multiple imputation produces valid estimates if the imputation model includes the variables that predict missingness.

MNAR

Missingness depends on the unobserved value itself. No standard method fully corrects this; sensitivity analyses are essential.

The mechanism cannot be verified from observed data alone. Always run sensitivity analyses.

Before you clean

Codebooks and variable types

Why a data dictionary?

Without it you cannot define what “out-of-range” means for each variable. It also makes cleaning reproducible.

Include: name, type, valid range, coding, and source for every variable.

Continuous Categorical Binary Count

Variable type determines which checks apply and which statistical operations are valid.

Carry forward

Three things to take into the next section

  • Errors enter at every stage: collection, entry, storage, and analysis all need checks.
  • Mechanism before method: decide which mechanism (MCAR, MAR or MNAR) is plausible, state it as an assumption, and then choose how to handle missingness.
  • No codebook, no cleaning: the dictionary defines what “valid” means for every variable.

Overview

A structured workflow takes raw data from collection to analysis-ready files. This lesson concentrates on two phases of that workflow: cleaning the data (detecting errors, handling missing values and deciding what to do with outliers) and producing descriptive analyses (the numerical summaries, tables and plots that every analysis report opens with). This section lays the groundwork for both. It traces where errors enter the data pipeline, names the common types of error and the coding faults that can be spotted in a raw file, sets out the three mechanisms by which values go missing and what each means for an analysis, and describes the variable types recorded in a data dictionary. It closes by introducing the course dataset and its codebook and taking a first look at the raw file in R.

Learning Objectives

  • 1.1 Trace where errors enter the collection → entry → storage → analysis pipeline.
  • 1.2 Distinguish between common error types (transcription, coding, logical inconsistency, out-of-range and duplicate; random vs. systematic; structural vs. value-level), recognise coding faults in a raw extract, and state how each is corrected.
  • 2.1 Classify missing-data mechanisms (MCAR, MAR, MNAR), justify the classification, and explain what each implies for a complete-case analysis.
  • 5.1 Describe variable types in epidemiologic data and the operations each type supports.

1.1 Where Errors Enter the Data Pipeline

Before any statistical analysis can begin, data travel through a pipeline of four stages: collection, entry, storage and analysis. Errors can be introduced at every stage, and knowing where a problem arose is the first step toward finding and correcting it. At the collection stage, errors may arise from poorly worded survey items, interviewer bias, instrument malfunction or participant misunderstanding. During data entry, when values are copied from a form or device into a file, transcription mistakes and coding errors are common. Storage introduces risks of data corruption, version-control problems and format incompatibilities. Finally, at the analysis stage, incorrect variable coding, inappropriate transformations and software errors can compromise results.

Data quality has several dimensions. Wang and Strong (1996) asked data users what quality meant to them and found many dimensions beyond accuracy. Four of them matter at every cleaning step and appear in the walkthrough above: accuracy (the values are correct), completeness (the values are present), timeliness (the values are current enough for the question) and consistency (the values are recorded in the same format and agree with each other). A fifth, relevance, asks whether the data suit the research question at all. The principle that ties the pipeline and these dimensions together is worth committing to memory.

The Quality of an Analysis Cannot Exceed the Quality of Its Data

The quality of your analysis can never exceed the quality of your data. Careful attention to data quality at each stage of the pipeline is a prerequisite for valid epidemiologic inference.

The errors introduced at these stages take several recognisable forms, and naming them is the subject of the next subsection.

1.2 Types of Data Errors and How They Are Corrected

1.2.1 Screening, Diagnosing and Editing

Errors in epidemiologic datasets take many forms. Van den Broeck, Argeseanu Cunningham, Eeckels, & Herbst (2005) describe data cleaning as a three-stage process of screening, diagnosing and editing that should be planned into the study design rather than improvised after collection. Screening means looking systematically for values that seem odd, such as an age of 220 or an empty cell where an answer was expected. Diagnosing means deciding what each odd value is: an error, a true but unusual value, or a missing-data code. Editing means acting on the diagnosis by correcting the value from the source document, setting it to missing, or leaving it unchanged, and recording the decision. Planning these steps before data collection allows the study team to build some of the checks into the data-entry system itself.

Screening is easier when the analyst knows what to look for. Five types of error account for most of what turns up: transcription errors, coding errors, logical inconsistencies, out-of-range values and duplicate records. Click each card below to see how each type arises, an example of it, and the check that usually finds it.

Transcription ErrorsClick to explore
Coding ErrorsClick to explore
Logical InconsistenciesClick to explore
Out-of-Range ValuesClick to explore
Duplicate RecordsClick to explore

Each card names an error by what it looks like in the file, and each points to a different check: double data entry for transcription errors, comparison with the codebook for coding errors, cross-field rules for logical inconsistencies, range checks for out-of-range values, and matching on key identifiers for duplicates.

1.2.2 Random, Systematic, Structural and Value-Level Errors

The five error types describe what an error looks like in the file. Two further distinctions describe how an error behaves, and both affect what the analyst can do about it.

Random versus systematic error. A random error is an error whose direction cannot be predicted. By this we mean that any one value is as likely to be recorded too high as too low: a reader misjudges a blood pressure gauge by a few mmHg, sometimes upward and sometimes downward. Because the errors scatter in both directions, they add noise to the data. This widens the spread of the values and makes estimates less precise, so that a repeated study would give more variable answers. When many values are averaged, the upward and downward errors tend to cancel out, so the mean is not pushed in one direction. Random error is not harmless, however. When it affects an exposure, it blurs the difference between people with high and low exposure, and as a result the estimated association between exposure and outcome is weakened toward no effect. For example, if some heavy drinkers are recorded as moderate drinkers and some moderate drinkers as heavy drinkers, the two groups look more alike than they really are, and any difference in their outcomes shrinks. In addition, random error matters whenever a measurement is compared with a cut-off. If high blood pressure is defined as a systolic reading of 140 mmHg or more, a true reading of 138 recorded as 142 moves a person above the cut-off, so individual people are misclassified and the proportion above the cut-off changes, even though the mean is largely unaffected.

A systematic error, by contrast, pushes values in the same direction every time, as with a miscalibrated scale that adds 2 kg to every weight. Systematic errors do not cancel out when values are averaged, so they produce bias, meaning an estimate that is consistently too high or too low. Collecting more data does not help, because every new value carries the same error. The two kinds of error are also found in different ways. Range checks catch some random errors, such as an impossible value, whereas a systematic error usually leaves values that look plausible and can be found only by comparison with an outside source, such as a recalibrated instrument.

Structural versus value-level error. A structural error affects the layout of the file: a variable stored as text when it should be a number, two variables merged into one column, a record that appears twice, or a missing-data code such as -9 mixed in with real values. A value-level error affects a single cell, such as one blood pressure entered as 1,200. The distinction matters for the order in which problems are fixed. Structural problems are usually fixed first, because value-level checks such as range checks give misleading answers when a column has the wrong type or contains codes. For example, if blood pressure is stored as text because one cell reads "not measured", R treats the whole column as text, so summary() reports no minimum or maximum and a range check cannot work as intended, and if a missing-data code of 999 sits among real readings, a range check flags every coded cell as an impossible value rather than as a missing one. Once the structure is correct, each remaining odd value can be judged on its own.

1.2.3 Coding Faults in a Raw File

Many structural problems are coding faults: they are built into a file by the way its values were recorded, and they can be spotted by reading a few raw rows beside the data dictionary. The table of coding faults below lists five that appear often in survey files. Read each row from left to right, because a complete answer about a fault always has two halves: what the problem is and why it matters, and the correction that best practice calls for.

Coding faults. Five common coding faults in a raw survey file, why each matters, and how each is corrected.

Coding faultExampleWhy it mattersCorrection
A missing-value code that could be a real valueNumber of children recorded as 9 for "don't know"A family with nine children cannot be told apart from a non-response, and the code is averaged as if it were a count.Use a code that can never be a real response, such as −999, declare it in the data dictionary, and convert it to NA in the analysis script.
A compound code (two pieces of information in one variable)smoke_alc = 23, where the first digit is smoking status and the second is drinking frequencyThe variable cannot be summarised or checked directly, and a missing code such as 99 splits into two values of 9 that look like real but out-of-range answers.Split it into two variables, one piece of information each, with its own valid range and missing code.
Text in a numeric fieldHours of exercise recorded as "none" or "unknown"R reads the whole column as text (chr), so no mean or range check works, and as.numeric() turns every text entry into NA.Record each entry as the number it means ("none" becomes 0) or as missing ("unknown" becomes NA), and keep the column numeric.
Inconsistent or ambiguous date formats03/04/2025, 4 Apr 25 and 2025-04-05 in one column03/04/2025 can mean 3 April or 4 March, and ages or durations calculated from mixed formats are wrong.Store dates as a date variable in one unambiguous format, YYYY-MM-DD, and state the format in the data dictionary.
Inconsistent missing valuesSome unanswered items left blank and others coded 99A blank may mean "not asked", "refused" or "lost", and some codes are missed when missing values are converted.Use one documented convention for every variable, never a blank, and return incomplete forms to the source for completion where possible.

Every correction is made in a script that reads the untouched raw file and writes a new version of it, never by editing the raw file by hand, so that each change can be checked and repeated. The next section carries out these steps in R. The errors described so far affect values that are present in the file; the next subsection turns to values that are absent.

2.1 Missing-Data Mechanisms

Missing data are ubiquitous in epidemiologic research, and the mechanism by which values become missing matters as much as how many are missing, because it decides whether an analysis of the people who remain gives valid answers. A missing-data mechanism is the process that decides which values end up missing. By this we mean the answer to the question "why is this value absent, and is the reason connected to anything about the person?" Rubin (1976) identified three mechanisms, which differ in what the missingness depends on. Data are missing completely at random (MCAR) when missingness is unrelated to anything about the people, observed or not; the people with missing values are then simply a random subset of the sample. Data are missing at random (MAR) when missingness depends only on variables that are recorded in the file. For example, younger people may skip a question more often than older people, but among people of the same age, whether the answer is missing has nothing to do with what the answer would have been. Data are missing not at random (MNAR) when missingness depends on the missing value itself, as when people with the most severe symptoms are the least likely to report them. The names are easy to confuse, because "missing at random" does not mean that the missingness is unrelated to everything; it means that the missingness is random once the recorded variables are taken into account. Click each card below to see an example of each mechanism and what it implies for a complete-case analysis, which analyses only the people with no missing values.

MCARClick to explore
MARClick to explore
MNARClick to explore

The cards show that the three mechanisms differ in what the missingness depends on, and that difference is what a classification has to justify.

Classifying a described pattern. Two questions settle most cases. First, is the missingness related to anything about the people, observed or not? If it arises from something unrelated to them, such as an equipment failure on days or in households assigned without regard to who the participants were, the pattern is MCAR, and a complete-case analysis remains unbiased but is less precise because the sample is smaller. Second, if the missingness is related to the people, is it fully explained by variables recorded in the file, such as age or survey format? If so, the pattern is MAR: a complete-case analysis can be biased, because the people who remain differ from those who are lost, and multiple imputation that uses the variables that predict missingness can give valid estimates. If instead the missingness depends on the missing value itself, for example people with the worst symptoms declining to report them, the pattern is MNAR: a complete-case analysis is biased in a direction that can usually be reasoned out (here, symptoms would be understated), imputation under the MAR assumption does not correct it, and sensitivity analyses are needed. A good justification is one sentence that names what the missingness depends on.

One caution applies to every classification, and it is a common source of exam errors.

The Mechanism Is an Assumption

The missing data mechanism is an assumption that cannot be verified from the observed data alone. You should always conduct sensitivity analyses to assess how your results might change under different assumptions about the missing data mechanism.

The next section returns to these mechanisms when it compares methods for handling missing data. Recognising a value as missing or erroneous first requires a record of what each variable should contain, which is the role of the data dictionary and of the variable types it records.

5.1 Variable Types and the Data Dictionary

5.1.1 Data Dictionaries and Codebooks

A data dictionary (or codebook) is a structured document that describes every variable in a dataset: its name, label, type, valid range, coding scheme, and source. Maintaining a comprehensive data dictionary is essential for:

  • Ensuring consistency across data entry operators
  • Facilitating communication among research team members
  • Supporting reproducibility when analyses are revisited months or years later
  • Identifying and resolving coding discrepancies

5.1.2 Variable Types in Epidemiologic Data

Among the entries in a data dictionary, the type of each variable matters most for analysis, because correctly classifying variables is fundamental to choosing appropriate cleaning strategies and analytic methods. Epidemiologic data contain four main types: continuous, categorical (nominal or ordinal), binary and count variables. Click each card below to see what each type is, with examples, and the cleaning checks that suit it.

ContinuousClick to explore
CategoricalClick to explore
BinaryClick to explore
CountClick to explore

The type decides which operations make sense. Means, standard deviations and differences suit continuous variables. For a categorical variable stored as numbers, the numbers are only labels: if blood type is coded 1 = A, 2 = B, 3 = AB and 4 = O, a mean blood type of 2.4 describes no real blood type, so a nominal variable is summarised by the count and percentage in each category, and an ordinal variable by the same counts kept in their natural order. A binary variable is summarised as the number and percentage with the characteristic, and a count variable by its mean and variance or its median.

5.1.3 The Course Dataset and Its Codebook

The R activities in this lesson use phaa_survey.csv, a simulated cross-sectional survey of 800 adults. Table 2.1 is a codebook excerpt for the variables this lesson works with most. A hard limit marks values that cannot be correct for this survey; values beyond it are treated as errors. A soft limit marks values that are possible but unusual; values between a soft and a hard limit are flagged for investigation, checked against the source where possible, and kept unless they are shown to be wrong. The limits are the rules applied in this lesson.

Table 2.1. Codebook excerpt for the variables of phaa_survey.csv used most in this lesson, with hard limits, soft limits and missing-data codes.

VariableType and unitsValid values (hard limits)Soft limits (flag and check)Missing code
idText identifierP0001 to P0800, each appearing onceNoneNever missing
ageContinuous; whole years18 to 100 (only adults were eligible)Above 90Blank cell
systolic_bpContinuous; mmHg, whole numbers80 to 220Below 90 or above 180Blank cell
bmiContinuous; kg/m², one decimal13 to 60Below 16 or above 50Blank cell
smokerBinary; text"Yes" (current smoker) or "No"NoneBlank cell
dep1 to dep7Ordinal questionnaire items1 (Never) to 5 (Always); a higher value means a more frequent depressive symptomNoneBlank cell
incomeOrdered categories; text"<25k", "25-50k", "50-75k", "75-100k", ">100k"NoneBlank cell

In this file a missing value is an empty cell, which R reads as NA (not available). Many real files instead store codes such as -9 (refused) or 999 (not asked); the codebook is where such codes are declared, and the next section shows how to convert them before any analysis. A full codebook would also record each variable's source, such as the questionnaire item or the measurement protocol; in this simulated file every variable comes from a single survey record per participant.

The codebook tells the analyst what each variable should contain; checking what the file actually contains is done in R, which needs a little preparation first.

5.1.4 Getting Ready to Run R and Reading R Code

The R boxes in this lesson read the course file directly from the course website, so no data file needs to be saved before they are run. The setup steps below are needed once, before the first R box: installing the add-on packages and running each activity's code from a script.

Setup Before the First R Box

Each R box in this lesson can be run in RStudio or any R console on a computer with an internet connection. No data file needs to be downloaded first: each activity reads the course data directly from the course website, for example with read.csv("https://heal-learn.ca/r-activities/data/phaa_survey.csv"), so R does not need to be pointed at a folder. Two preparations are needed once, before the first box.

  1. Install the add-on packages used in this lesson by running install.packages(c("psych", "tableone")) once; the optional multiple-imputation panel in the next section also needs install.packages("mice"). A package is a bundle of extra functions; installing copies it to the computer, and library(psych) loads it at the start of each new R session.
  2. Download the Code file from each R box and open it in RStudio, or open a new R script (File > New File > R Script) and copy the code from the box into it. Each Code file reads the data it needs from the website, so each activity runs on its own; within an activity the lines are run in order, because later lines use objects created by earlier ones.

The Data link in each R box gives a copy of the file for anyone who wants to open it in a spreadsheet. A saved copy is read only from R's working directory, the folder R is currently pointed at: getwd() prints that folder, and in RStudio the menu Session > Set Working Directory > Choose Directory changes it.

A line of code runs when the cursor is on it and Ctrl+Enter (Cmd+Return on a Mac) is pressed, and the result appears in the Console pane. Selecting several lines and pressing the same keys runs them together, and the Source button runs the whole script from top to bottom. Running one line at a time allows each result to be compared with the annotated output on this page.

The code shown on this page is the starter code for every activity and can be copied directly. The answer-key script linked in each activity box contains the same code together with worked answers, and it becomes available after the responses to that box are saved.

For a longer demonstration, open the narrated R walkthrough below, which applies the cleaning and descriptive steps of this lesson to a real survey, the Canadian Social Connection Survey.

Learn to do this in R

Narrated R walkthrough: Data Cleaning and Descriptive Statistics in R

This walkthrough applies the steps of this lesson to the Canadian Social Connection Survey, one line at a time. It keeps the rows and columns an analysis needs, explores each variable before changing it, recodes and checks every recode, and then describes the data with summary statistics, plots, stratified tables and a Table 1.

Open the Data Cleaning and Descriptive Statistics in R walkthrough

R code is a sequence of instructions. Most lines either create an object (a named result stored in memory) or ask R to display something. Table 2.2 lists the symbols and terms that appear throughout the R code of this lesson; refer back to it whenever a line of code is unclear.

Table 2.2. Symbols and terms that appear in the R code of this lesson, each with an example and its meaning.

Symbol or termExampleWhat it means
<-x <- c(4, 7, 9)The arrow stores the result on its right under the name on its left. The example creates an object called x that holds three numbers.
c()c("age", "bmi")The function c() combines values into a vector, an ordered list of values of the same type.
Functions and argumentsmean(x, na.rm = TRUE)A function is a named command followed by parentheses. The values inside the parentheses are its arguments, and name = value sets an option.
$phaa$ageThe dollar sign picks one column out of a data frame (R's name for a table of rows and columns). phaa$age is the age column of the data frame phaa.
[ ]phaa$age[3]
phaa[1:5, c("id", "age")]
Square brackets select elements. For a single column, [3] picks the third value. For a data frame, the part before the comma picks rows and the part after the comma picks columns, so the second example shows the ID and age of the first five people.
== < > <= >= !=phaa$age < 18Comparisons return TRUE or FALSE for each value. A double equals sign tests whether two values are equal, and != tests whether they differ.
| and &phaa$age < 18 | phaa$age > 100The vertical bar means "or" and the ampersand means "and". The example is TRUE for any age below 18 or above 100.
!!is.na(phaa$bmi)The exclamation mark turns TRUE into FALSE and FALSE into TRUE, so the example is TRUE wherever BMI is present.
NAphaa$bmi[phaa$bmi > 60] <- NANA is R's code for a missing value. The example replaces every BMI above 60 with a missing value.
na.rm = TRUEmean(phaa$age, na.rm = TRUE)Most summary functions return NA when any value is missing. The argument na.rm = TRUE (remove NAs) tells the function to leave missing values out.
~boxplot(systolic_bp ~ smoker, data = phaa)The tilde reads as "by" or "depending on". The example draws systolic blood pressure separately for each smoking group.
## a note for readersEverything after # on a line is a comment for human readers, and R ignores it.
library()library(psych)The function loads an installed package so that its functions can be used in the current session.

5.1.5 A First Look at the Raw File

Screening begins by reading the raw file and comparing what arrived with the codebook in Table 2.1, which brings together the ideas of this section: the variable types, the error types and coding faults, and the missing values. A few commands do most of this work: dim() reports the number of rows and columns, str() (structure) lists each variable with its type and first few values, summary() gives the minimum, quartiles, mean and maximum of numeric variables, table() counts the values of a categorical variable, and colSums(is.na()) counts the missing values in each column (is.na() marks each missing cell as TRUE and colSums() adds up the TRUEs in each column). Activity 2.1 practises this first screen of phaa_survey.csv one part at a time and asks what each part of the output shows and what to do with it; run each part along with the box and compare your output with the notes.

R Activity 2.1: a first look at the raw survey file

Text version of the activity: the code of the run-along script, its console output and notes on reading it.

# Blank cells and the text "NA" become missing values (NA)
phaa <- read.csv("https://heal-learn.ca/r-activities/data/phaa_survey.csv",   # course data, read from the website
                 stringsAsFactors = FALSE, na.strings = c("", "NA"))
dim(phaa)                 # rows (people) and columns (variables)
Console output
[1] 800 27

The file has 800 rows, one per participant, and 27 columns, one per variable, which matches the codebook. A different count would suggest an incomplete download or a different version of the file.

str(phaa)                 # each variable's type and first few values
table(phaa$income, useNA = "ifany")   # income bands, stored as text
Console output
'data.frame': 800 obs. of 27 variables: $ id : chr "P0001" "P0002" "P0003" "P0004" ... $ age : int 49 46 40 55 56 43 49 39 59 64 ... $ gender : chr "Woman" "Man" "Woman" "Man" ... $ ethnicity : chr "White" "White" "East Asian" "White" ... $ education : chr "Some college" "High school" "Some college" "High school" ... $ income : chr "25-50k" "50-75k" "50-75k" "50-75k" ... $ region : chr "Surrey" "Other" "Surrey" "Richmond" ... $ smoker : chr "No" "No" "No" "No" ... $ phys_act_min : int 196 256 0 253 130 221 355 180 84 236 ... $ discrimination_score: int 7 3 4 5 0 8 3 12 4 5 ... $ social_support_score: int 30 19 32 20 32 27 31 19 31 22 ... $ dep1 : int 5 1 1 4 1 2 5 2 2 4 ... $ dep2 : int 4 3 3 2 2 4 5 3 2 5 ... $ dep3 : int 4 1 5 2 2 2 5 3 4 4 ... $ dep4 : int NA 1 2 2 1 2 5 5 1 4 ... $ dep5 : int 4 2 3 4 1 2 5 3 3 5 ... $ dep6 : int 5 1 3 3 2 1 5 2 1 5 ... $ dep7 : int 5 1 2 3 3 4 5 5 4 5 ... $ anx1 : int 5 2 3 5 2 1 5 4 2 1 ... $ anx2 : int 4 1 2 5 2 3 4 NA 3 1 ... $ anx3 : int 4 1 1 2 2 1 4 2 3 1 ... $ anx4 : int 4 2 1 5 1 3 5 2 3 1 ... $ anx5 : int 2 1 1 5 1 1 3 4 3 3 ... $ bmi : num 21.8 20.8 20.3 21.5 21.3 15 20 23.9 23 21.4 ... $ systolic_bp : int 111 116 122 104 114 114 101 104 101 128 ... $ diastolic_bp : int 70 73 78 65 71 78 71 75 71 88 ... $ hypertension : chr "No" "No" "No" "No" ... <25k >100k 25-50k 50-75k 75-100k <NA> 94 135 159 207 170 35

The label chr means text (character), int means whole numbers and num means numbers that can have decimals. Every variable has the storage type the codebook leads one to expect; a numeric variable listed as chr would signal text, such as "unknown", hiding in the column. Storage is not the same as type: income is an ordinal variable stored as text, so table() lists its bands in alphabetical order, and the questionnaire items dep1 to dep7 are ordinal items stored as whole numbers.

summary(phaa[, c("age", "systolic_bp", "bmi", "dep1")])
phaa[phaa$age < 18 | phaa$age > 100, c("id", "age")]       # valid: 18 to 100
phaa[phaa$systolic_bp < 80 | phaa$systolic_bp > 220,       # valid: 80 to 220
     c("id", "systolic_bp", "diastolic_bp")]
phaa[phaa$bmi < 13 | phaa$bmi > 60, c("id", "bmi")]        # valid: 13 to 60
Console output
age systolic_bp bmi dep1 Min. : -3.00 Min. : 60.0 Min. :15.00 Min. :1 1st Qu.: 36.00 1st Qu.:100.0 1st Qu.:18.50 1st Qu.:2 Median : 45.00 Median :109.0 Median :20.70 Median :3 Mean : 45.31 Mean :108.4 Mean :20.86 Mean :3 3rd Qu.: 54.00 3rd Qu.:116.0 3rd Qu.:23.20 3rd Qu.:4 Max. :220.00 Max. :300.0 Max. :95.00 Max. :5 id age 33 P0033 -3 612 P0612 220 id systolic_bp diastolic_bp 17 P0017 300 76 245 P0245 60 63 id bmi 78 P0078 95

The middle of each distribution (quartiles and median) looks ordinary, but five values break the hard limits of Table 2.1: ages of -3 (P0033) and 220 (P0612), systolic blood pressures of 300 (P0017) and 60 (P0245, whose diastolic pressure of 63 is higher than its systolic pressure) and a BMI of 95 (P0078). The next section diagnoses them and sets them to missing.

table(phaa$smoker, useNA = "ifany")   # counts for a text variable
sum(duplicated(phaa$id))              # IDs that appear more than once
colSums(is.na(phaa))                  # missing values in each column
Console output
No Yes 664 136 [1] 0 id age gender 0 0 0 ethnicity education income 0 0 35 region smoker phys_act_min 0 0 28 discrimination_score social_support_score dep1 0 20 0 dep2 dep3 dep4 0 0 15 dep5 dep6 dep7 0 0 0 anx1 anx2 anx3 0 15 0 anx4 anx5 bmi 0 0 0 systolic_bp diastolic_bp hypertension 0 0 0

Smoking status takes only the two codes in the codebook, "No" (664) and "Yes" (136), with no missing values, so no recoding is needed; variants such as "yes" or "Y" would appear as extra categories. No ID appears twice. Five variables have missing values: income (35), physical activity (28), social support (20), and the items dep4 (15) and anx2 (15). These counts are the starting point for the missing-data decisions in the next section.

Files for this activity Data Code Answer key

R Reflect on what you just ran

Answer each question from your own output, as on the midterm exam: find the value in the output, say what it means, and say what to do about it.

1. Using the output of str() and table() and the codebook in Table 2.1, state the type of age, smoker, income and dep1, and how R stores each. For household income, type round(100 * prop.table(table(phaa$income)), 1) at the end of the script and run it to print the percentage in each band; then state the measure of central tendency and the measure of spread you would report, and justify the choice with evidence from the output.

Model answerAge is continuous (whole years) and is stored as whole numbers (int); smoker is binary and is stored as text (chr, "Yes" or "No"). Income is ordinal (five ordered bands) but is stored as text, so R does not know the order and table() lists the bands alphabetically, with >100k second. The item dep1 is also ordinal (1 Never to 5 Always) but is stored as whole numbers, so R would average it as if the steps between answers were equal. The storage type therefore shows how R will treat a column, not what its values mean, so the type comes from the data dictionary; a numeric variable listed as chr would signal text hiding in it. For income, report the count and percentage in each band, in order, among the people with income data: the counts come from line 14 and the percentages from your prop.table() line (under 25k: 94, 12.3%; 25-50k: 159, 20.8%; 50-75k: 207, 27.1%; 75-100k: 170, 22.2%; over 100k: 135, 17.6%), with the modal band, 50-75k, as the typical value. Income is recorded in ordered categories, so a mean and standard deviation have no meaning; the spread is shown by the percentage distribution, and the 35 missing values are reported.

2. The output shows values in three variables that break the hard limits in Table 2.1. Identify each faulty value with its ID, name the type of error it most likely is, and state the stage of the data pipeline where it most likely entered. Then state how each should be corrected, according to best practice.

Model answerAge is -3 for P0033 and 220 for P0612 (valid range 18 to 100), systolic blood pressure is 300 for P0017 and 60 for P0245 (valid range 80 to 220), and BMI is 95 for P0078 (valid range 13 to 60). Each is an out-of-range value, a value-level error in a single cell; values such as 220 and 95 look like typing slips (an extra or transposed digit), so they most likely entered at data entry as transcription errors, although a faulty instrument at collection is possible for a blood pressure. P0245 is also a logical inconsistency, because its diastolic pressure (63) is higher than its systolic pressure (60). The quartiles and medians look ordinary, so these are isolated random errors rather than a systematic shift; a range check cannot catch a wrong value that looks plausible, such as 68 entered as 86, which needs double data entry or a check against the form. Each value is checked against the source record (the original form) and corrected if the true value can be recovered; otherwise it is set to NA in the cleaning script, keeping the row and the person's other variables. Every correction is made in a script that reads the untouched raw file and is recorded in the cleaning log (the rule, the IDs and the number of values changed); the raw file is never edited by hand.

3. Run your own code: at the end of the script, type tapply(phaa$age, is.na(phaa$income), median) and run it. Three possible explanations for the 35 missing incomes are described below. For each, classify the mechanism as missing completely at random, missing at random or missing not at random; justify the classification in one sentence; and state the consequence of analysing complete cases only. (i) For one batch of paper forms, the copier left blank the page that held the income question, and the forms were copied in random order. (ii) Income is missing more often among younger and less educated respondents, both recorded in the file, and shows no further relationship to income once age and education are accounted for. (iii) Respondents with the highest incomes chose not to answer the income question. Finally, what does your output show, and can it tell you which explanation is true?

Model answer(i) Missing completely at random: the forms were copied in random order, so the loss is unrelated to any characteristic of the respondents, observed or unobserved; a complete-case analysis remains unbiased, and the cost is a smaller sample and wider confidence intervals. (ii) Missing at random: the probability that income is missing depends on age and education, which are observed, and not on income itself once they are accounted for; a complete-case analysis can be biased, because respondents who report income differ from those who do not, and multiple imputation that uses age and education can recover valid estimates. (iii) Missing not at random: the probability that income is missing depends on the unobserved income itself; a complete-case analysis would understate income (the highest incomes are the ones missing), imputation under the missing-at-random assumption does not correct this, and sensitivity analyses are required. Your tapply() line gives a median age of 45 among those who reported income (FALSE) and 43 among those who did not (TRUE), so the missingness is barely related to age. The output cannot show which explanation is true: a clear difference would be evidence against MCAR, but no comparison of observed values can rule out MNAR, because it depends on the values that are missing; the mechanism is an assumption, so state it and check conclusions with sensitivity analyses.
Saved.

The first screen in Activity 2.1 found each kind of problem described in this section: values that break the codebook's limits, columns whose storage type must match the codebook, and gaps whose mechanism has to be judged before they are handled. This section has traced where errors enter the data pipeline, classified types of error and missing-data mechanisms, and described the variable types recorded in a data dictionary. The key takeaways below summarise each of these points, the knowledge check that follows reviews them, and the next section sets out the cleaning workflow that addresses the problems found.

Key Takeaways

  • 1.1 Data travel through four stages before you analyse them (collection, entry, storage and analysis), and errors can enter at every one. A poorly worded question, interviewer bias or a faulty instrument causes problems at collection; transcription slips and wrong codes happen at entry; corrupted files, version mix-ups and format incompatibilities arise in storage; and incorrect recodes, inappropriate transformations or software errors appear at the analysis stage. This is why data quality is checked throughout a project rather than with one final glance at the finished file: the quality of your analysis can never exceed the quality of your data. When an example describes a problem, name the stage where it most likely entered, because that tells you where the check belongs and where a correct value might be recovered, such as the original paper form for an entry error.
  • 1.2 Errors can be named by what they look like in the file: transcription errors (transposed digits, a shifted decimal point), coding errors (a wrong or reversed code), logical inconsistencies (fields that contradict each other, such as a diagnosis dated before birth), out-of-range values and duplicate records. They can also be described by how they behave: random errors add noise and reduce precision, systematic errors push values the same way and cause bias, structural errors affect a whole column or the layout of the file, and value-level errors affect a single cell (fix structural problems first, because range checks mislead when a column holds text or codes). When you are given a few raw rows and a data dictionary, read each column against its entry and look for the common coding faults: a missing code that could be a real value (9 for "don't know" in a count of children), a compound code that stores two pieces of information in one variable, text such as "none" in a numeric field, mixed or ambiguous date formats (is 03/04/2025 3 April or 4 March?), and blanks mixed with missing codes. For each fault, state the correction as well as the problem: an impossible missing code such as −999 declared in the dictionary and converted to NA, one variable for each piece of information, the number the text means ("none" becomes 0), dates stored as YYYY-MM-DD, and one documented missing-value convention. Make every correction in a script that reads the untouched raw file, never by editing the raw file by hand.
  • 2.1 Why values are missing matters as much as how many are missing, and Rubin's three mechanisms give you the words for it. Data are missing completely at random (MCAR) when the missingness has nothing to do with the people, observed or not, as when a freezer fails at random; missing at random (MAR) when it depends only on variables recorded in the file, such as younger people skipping the income question; and missing not at random (MNAR) when it depends on the missing value itself, as when people with the worst symptoms decline to report them. To classify a described pattern, ask two questions: is the missingness related to anything about the people, and if so, is it fully explained by variables in the file? Then answer in three parts: the label, a one-sentence justification that names what the missingness depends on, and the consequence for a complete-case analysis, which is unbiased but less precise under MCAR, possibly biased under MAR (where multiple imputation using the variables that predict missingness can give valid estimates), and biased under MNAR in a direction you can usually reason out (with the worst symptoms missing, symptoms would be understated). Because the mechanism cannot be confirmed from the observed data, state it as an assumption and check your conclusions with sensitivity analyses.
  • 5.1 Variables come in different types depending on what their values represent: continuous (any value in a range, such as blood pressure or BMI), categorical, which may be nominal (no order, such as blood type) or ordinal (a meaningful order, such as disease severity or income bands), binary (exactly two categories) and count (non-negative whole numbers, such as hospital visits). You cannot always tell a variable's type from how its values are stored: ordinal questionnaire items such as dep1 to dep7 are stored as numbers, binary variables may be stored as 0 and 1, and a numeric variable may arrive as text, so check the data dictionary, which records each variable's type, valid values and missing codes. In R, str() shows how each column is stored (chr for text, int for whole numbers, num for numbers with decimals); this tells you how R will treat the column, not what its values mean, and a numeric variable listed as chr is a sign that text is hiding in it. Once you know the type, match the check to it: range and shape checks for continuous variables, frequency tables for unexpected categories or misspellings, exactly two valid values for binary variables, and no negative or implausibly large values for counts.
Knowledge check: this section

1. A research assistant types paper questionnaires into a spreadsheet. On one form the participant wrote a weight of 68 kg, but the spreadsheet shows 86 kg. At which stage of the data pipeline did this error enter, what type of error is it, and which check is most likely to catch it?

The form was correct, so the error did not enter at collection, and swapped digits are a transcription error, not a coding error. A range check would also miss it, because 86 kg is an ordinary weight.
Nothing suggests the stored file was damaged; the wrong value was typed when copying from paper. A backup would simply restore the same mistyped value.
Copying from paper is the entry stage, and swapping two digits is a transcription error. Because the wrong value is plausible, only comparison with an independent second entry, or the form itself, reveals it.
The stage and error type are right, but a range check flags only values outside its limits. A plausible wrong value such as 86 kg passes it, so double entry is the check that catches it.
The answer is (c). The information needed is where the mistake happened (the form was right and the copy was wrong) and whether the wrong value is plausible. Copying from paper into a computer is the entry stage, and swapping two digits while copying is a transcription error; because 86 kg is an ordinary weight, only a comparison with an independent second entry, or with the form itself, reveals it. Option (d) names the right stage and type, but a range check flags only values outside its limits, and 86 kg lies well inside them. (Section 1, 1.1 Where Errors Enter the Data Pipeline and 1.2.1 Screening, Diagnosing and Editing, Transcription Errors card; Objectives 1.1 and 1.2)

2. A community organisation sends the raw extract and data dictionary below. A colleague proposes four corrections. Which one follows best practice?

id   sleep_hrs  smk_alc  start_date   exercise_min
101  7          21       2025-02-03   30
102  9          99       03/02/2025   none
103  6.5        3        4 Feb 25     45

Data dictionary as supplied
  sleep_hrs     hours of sleep on a typical night; 9 = refused
  smk_alc       first digit = smoking (1 never, 2 former, 3 current)
                second digit = alcohol use (1 none, 2 monthly, 3 weekly)
                99 = missing
  start_date    date of first visit
  exercise_min  minutes of exercise on a typical day
A refusal coded 9 cannot be told apart from a person who sleeps 9 hours. A code that no real answer can take, declared in the dictionary and converted to NA in the script, keeps the two apart.
Converting text is needed, but "none" means 0 minutes, so as.numeric() would turn true zeros into missing values. The word should be recorded as the number it means.
A compound code should be split into a smoking and an alcohol variable, each with its own missing code. Keeping it whole leaves 99 to split into two 9s and leaves single-digit entries undecodable.
Corrections are made in a script that reads the untouched raw file, never by retyping and saving over the original. Dates are converted there to one unambiguous format such as YYYY-MM-DD.
The answer is (a). Read each value beside the dictionary and ask whether a code could be mistaken for a real answer: a refusal coded 9 cannot be told apart from a person who sleeps 9 hours, so refusals need a code that no real response can take, documented and converted to NA in a script. Each of the other proposals creates a new problem. "None" means 0 minutes, so as.numeric() would turn true zeros into missing values (b). smk_alc is a compound code that should be split into a smoking variable and an alcohol variable, each with its own missing code, because 99 would otherwise split into two values of 9 and the single digit 3 in row 103 cannot be decoded (c). Corrections are made in a script that reads the untouched raw file, with dates stored in one unambiguous format such as YYYY-MM-DD (03/02/2025 could mean 3 February or 2 March), never by hand (d). (Section 1, 1.2.3 Coding Faults in a Raw File, table of coding faults; Objective 1.2)

3. In a cohort of 1,600 adults, the 12-month depression score is missing for 15% of participants. Missingness is higher among younger participants and among those who live more than 50 km from the clinic, both recorded at baseline, and among people of the same age and distance it is unrelated to their depression score. Which classification and consequence are correct?

MCAR means missingness is unrelated to anything about the participants. Here it is related to age and distance, so the pattern is not MCAR and complete cases are not a random subset.
The missing variable being the depression score does not make the pattern MNAR. Among people of the same age and distance missingness is unrelated to depression, so recorded variables explain it.
The label is right, but recording the predictors of missingness helps only if they are used, for example in multiple imputation. A complete-case analysis ignores them and can still be biased.
Missingness depends on recorded age and distance, and not on depression once these are accounted for, which is MAR. Complete cases then differ from the full cohort, while imputation using those predictors can give valid estimates.
The answer is (d). Apply the two questions from Section 1: is the missingness related to anything about the participants (yes, to age and distance), and is it fully explained by recorded variables (yes, as the last clause of the stem states)? That makes the pattern MAR. A complete-case analysis keeps a group that is older and lives closer to the clinic than the full cohort, so its estimates can be biased, while imputation that includes the predictors of missingness gives valid estimates under the MAR assumption. Option (c) has the right label but the wrong consequence, because recording the predictors helps only if they are used, for example in multiple imputation; option (a) ignores the stated link with age and distance. (Section 1, 2.1 Missing-Data Mechanisms, "Classifying a described pattern"; Objective 2.1)

4. A data dictionary codes region as 1 = North, 2 = South, 3 = East and 4 = West. A colleague's summary table reports "Region, mean (SD): 2.4 (1.1)". What is the problem, and what should be reported instead?

Being stored as numbers does not make codes into measurements. The numbers only name regions, so their mean and SD describe no real region.
Region is nominal: its codes are labels with no order or distance, so a mean has no meaning. A categorical variable is described by the count and percentage in each category.
Region has no natural order, so it is nominal, not ordinal. A median code is as meaningless as a mean for categories that cannot be ranked.
A count variable records how many times something happens. Region is a categorical label, so mean-variance comparisons do not apply.
The answer is (b). The information needed is the variable type, which the dictionary shows: the numbers 1 to 4 only name four regions, and East is neither "more" than South nor a fixed distance from it. A mean of 2.4 therefore describes no real region, in the same way that a mean blood pressure makes sense but a mean city of residence does not. A categorical variable is described instead by how many people fall in each category, the frequency distribution that the Categorical card recommends checking. Option (c) is the tempting one, but region is nominal rather than ordinal: its categories have no natural order, so a median code has no more meaning than a mean. (Section 1, 5.1.2 Variable Types in Epidemiologic Data, Categorical card; Objective 5.1)

● Complete the knowledge check to continue.

Section 2

Data Cleaning Strategies

⏱ Estimated time: 50 minutes
Section 2 of 3

Data Cleaning Strategies

Systematic verification, outlier triage, missing-data methods, transformations, and the audit trail that ties it all together.

Verification

Three types of systematic checks

Range checks

Hard limits (impossible) and soft limits (unusual but possible). Define from the codebook before you look at the data.

Consistency checks

Cross-field logic: discharge after admission, age matches date of birth, skip patterns respected.

Cross-validation

Compare across data sources. Establish a reliability hierarchy when sources disagree.

Outliers

Three ways to find them; one rule for all of them

IQR method
\[ \color{#0B7B6B}{\text{Low}} = \color{#C2410C}{Q_1} - 1.5 \times IQR \quad \color{#1D4ED8}{\text{High}} = \color{#BE185D}{Q_3} + 1.5 \times IQR \]
Q1 25th percentile Q3 75th percentile Low / High outlier fences
Z-score
\[ \color{#0B7B6B}{z} = \frac{\color{#C2410C}{x} - \color{#6D28D9}{\bar{x}}}{\color{#1D4ED8}{s}} \quad \text{flag if } |z| > 3 \]
z standardized score x observed value x̄ sample mean s standard deviation

Visual inspection (box plots, histograms, scatter plots) should accompany any quantitative rule.

The rule: investigate before you delete. An extreme value may be a genuine clinical finding or a data-entry shift of one decimal place.

Tukey (1977): look at data before you model it.

Missing data

Methods ranked by how well they handle uncertainty

Listwise deletion

Simple but biased unless MCAR. Large cumulative losses in multivariable models.

Pairwise deletion

Uses all available cases per calculation. Preserves data but results may not fit together coherently.

Single imputation

Mean / median / mode. Underestimates variance; standard errors too narrow.

Multiple imputation

Widely used under MAR (Sterne et al., 2009). Generates 5–20 plausible datasets; pools results with Rubin’s rules.

Transformations

Addressing skew without discarding data

TransformationFormulaWhen to use
Log (natural)\(\ln(x)\) or \(\ln(x+1)\)Right-skewed; multiplicative relationships
Square root\(\sqrt{x}\)Count data; moderate skew
Reciprocal\(1/x\)Strongly skewed rates and times

Back-transform all results for reporting. A coefficient from a log-transformed outcome is multiplicative, not additive.

Recoding & strings

New variables from old; order from messy text

Recoding

Collapse categories (current + former → ever smoker); band continuous variables (BMI groups); derive quantities (person-years, age at onset). Every recode goes in the dictionary.

String cleaning

“Vancouver”, “vancouver”, “VANCOOUVER” → one standard value. Consistent case, trimmed whitespace, pattern rules, lookup tables, fuzzy matching.

Documentation

The audit trail is non-negotiable

Every cleaning decision needs to record:

  • What changed (original value and replacement)
  • Why (biological implausibility, logical contradiction, etc.)
  • Who and when

Reproducibility means another analyst can rebuild the cleaned dataset from the raw file using only your documentation.

Do not edit values by hand in a spreadsheet. Write a script. The script is the audit trail.Course guidance, paraphrasing the reproducibility principle in Wilkinson et al., 1999

Overview

The previous section named the kinds of errors that can enter a dataset and the mechanisms by which values go missing. This section sets out the cleaning workflow that deals with them. It begins with a systematic verification pass of range, type and consistency checks, moves on to outliers and the decision to keep, transform or exclude a flagged value, and then covers transformations, derived variables and the cleaning of text fields. It shows how every one of these decisions is documented so that the cleaned dataset can be rebuilt from the raw file, and it ends with the choice of a method for handling the missing values that remain.

Learning Objectives

  • 3.1 Run range, type, and consistency checks as a first systematic verification pass, distinguishing hard limits (impossible values) from soft limits (unusual but possible values).
  • 3.2 Identify outliers with the IQR and z-score rules and decide whether to keep, transform, or exclude them, with reasons recorded.
  • 3.3 Apply transformations and derive new variables without losing the audit trail back to the raw data.
  • 3.4 Document every cleaning decision, as a rule in a script and a cleaning log, so that another analyst could reproduce your final dataset.
  • 4.1 Choose an appropriate missing-data strategy (complete-case, single imputation, multiple imputation) given the plausible mechanism and the amount missing.

3.1 Systematic Verification: Range, Type and Consistency Checks

Data cleaning should follow a systematic, documented process. The goal is not to “fix” data arbitrarily but to identify and resolve genuine errors while preserving the integrity of valid observations. A first verification pass therefore applies rules that were decided before anyone looked at the data, taken from the codebook, rather than fixing whatever happens to catch the analyst's eye. A type check comes first: it confirms that each column is stored as the codebook expects, for example that age is a number rather than text, because, as the previous section explained, value-level checks give misleading answers when a column has the wrong type. In R, str() shows the storage type of every column. Three further kinds of check follow. Expand each panel below to see how range checks, consistency checks and cross-validation work, and note in particular the difference between a hard limit and a soft limit in the first panel.

Range Checks

Range checks verify that values fall within predefined valid or plausible bounds. For example, gestational age should be between approximately 20 and 44 weeks; human body temperature typically ranges from 35°C to 42°C.

Define both hard limits (biologically impossible values that should be flagged as errors) and soft limits (unusual but possible values that should be reviewed).

The two kinds of limit lead to different actions. A value beyond a hard limit is an error: it is corrected from the source document if possible and otherwise set to missing. A value between a soft limit and a hard limit is flagged for investigation: the analyst lists it, checks it against the source where possible, and keeps it unless it is shown to be wrong. The codebook excerpt in the previous section (Table 2.1) gives both kinds of limit for the course data.

Consistency Checks

Consistency checks examine whether values across multiple fields are logically compatible. Examples include verifying that discharge dates occur after admission dates, that a participant’s age is consistent with their reported date of birth, and that skip patterns in questionnaires are respected.

These checks often require domain knowledge about the relationships between variables. By this we mean that the rule cannot be read from any single column; the analyst has to know how the variables should relate to each other. For example, a skip pattern means that people who answer "never smoked" are not asked how many cigarettes they smoke per day, so a record with both "never smoked" and a value of 20 cigarettes contradicts itself, even though each answer is valid on its own. Writing such rules therefore depends on knowing the questionnaire and the subject matter as well as the data.

Cross-Validation

Cross-validation involves comparing data from multiple sources to verify accuracy. For example, comparing self-reported medication use with pharmacy dispensing records, or verifying diagnoses against medical chart reviews.

When discrepancies arise, establishing a hierarchy of data source reliability is essential for deciding which value to retain. A hierarchy of reliability is a ranking of the sources from the most to the least trustworthy for a particular variable, agreed in advance. For example, a study might decide that a diagnosis confirmed in the medical chart outranks a diagnosis reported by the participant, because the chart was recorded by a clinician at the time. Because the ranking is set before the discrepancies are examined, the analyst does not choose, case by case, whichever value happens to suit the expected result.

The panels describe three different questions: whether one value is possible (a range check), whether the values in one record agree with each other (a consistency check), and whether the file agrees with another source (cross-validation). A range check examines one variable at a time, so it cannot find a record in which each value is valid on its own but the values contradict each other; only a consistency check does that.

The hard limits are applied in part 3 of Activity 2.2, at the end of this section. This is the editing stage of the screening, diagnosing and editing sequence: the values were screened in Activity 2.1, diagnosed as impossible because they break a hard limit, and are then replaced, with each change listed first. In the course file two ages (−3 and 220), two systolic pressures (300 and 60 mmHg) and one BMI (95) break a hard limit; each is set to NA and its row is kept, so the person's other 26 variables remain available, whereas deleting the row would discard valid data and could bias the results if errors are not random. Values in the soft-limit zones are left as they are. The raw file itself is never overwritten: range-check rules change as an analyst learns more about the data, and an untouched raw file lets the cleaning be replayed from a known starting state.

Range checks identify values that cannot be correct, and the values they set to missing join those already missing in the raw file, which 4.1 returns to. Values that are possible but lie far from the rest of the data are outliers, the subject of the next subsection.

3.2 Identifying Outliers and Deciding What to Do With Them

3.2.1 The IQR and z-Score Rules

Outliers are observations that lie far from the bulk of the data. They may represent genuine extreme values, data errors, or observations from a different population. The visual tools used to spot them, including box plots, stem-and-leaf displays, and the broader practice of looking at data before modelling, trace back to Tukey (1977). Two numerical rules are in common use. The IQR rule flags values below Q1 − 1.5 × IQR or above Q3 + 1.5 × IQR (Equation 2.1), and the z-score rule flags values more than three standard deviations from the mean (Equation 2.2). Open each tab below to see the formula for each rule with a worked example, and the third tab for the visual checks that complement both.

The interquartile range (IQR) method defines outliers as values falling below Q1 − 1.5×IQR or above Q3 + 1.5×IQR, where Q1 and Q3 are the 25th and 75th percentiles, respectively.

Equation 2.1 gives the two bounds as formulas.

IQR outlier bounds
\[ \color{#0B7B6B}{\text{Lower}} = \color{#C2410C}{Q_1} - 1.5 \times \color{#6D28D9}{IQR} \qquad \color{#1D4ED8}{\text{Upper}} = \color{#BE185D}{Q_3} + 1.5 \times \color{#6D28D9}{IQR} \]Eq 2.1
An observation is an outlier if it falls below the lower bound (the 25th percentile minus 1.5 times the interquartile range) or above the upper bound (the 75th percentile plus 1.5 times that same range).

This method is robust to extreme values because it is based on percentiles rather than the mean and standard deviation. A statistic is called robust when a few unusual values have little effect on it. By this we mean that the quartiles depend only on where the values at the 25th and 75th percentiles fall in the sorted list, not on how far the most extreme values lie beyond them. For example, if the largest reading in a sample changed from 184 to 1,840 mmHg, Q1 and Q3, and therefore the fences, would not change at all, so the rule used to judge the extreme value is not distorted by that same value. Values beyond 3×IQR from the quartiles are sometimes termed “extreme outliers.”

Worked Example 2.1 shows where the quartiles, IQR and fences that R reports come from, using eleven systolic blood pressure readings, and Table 2.3 sets out what each quantity is.

Worked Example 2.1: IQR Fences for Eleven Readings

Eleven systolic blood pressure readings (mmHg), sorted from lowest to highest, are 96, 102, 106, 110, 112, 115, 118, 121, 125, 131 and 184.

Table 2.3. How the median, quartiles, IQR and fences of the eleven systolic blood pressure readings are formed.

StepCalculationResult
Median: the middle (6th) value96, 102, 106, 110, 112, 115, 118, 121, 125, 131, 184115
Q1: halfway between the 3rd and 4th values(106 + 110) / 2108
Q3: halfway between the 8th and 9th values(121 + 125) / 2123
IQR123 − 10815
Lower fence108 − 1.5 × 15 = 108 − 22.585.5
Upper fence123 + 1.5 × 15 = 123 + 22.5145.5
Values outside the fences184 is above 145.5184 is flagged

The reading of 184 mmHg is flagged for investigation. It also lies beyond Q3 + 3 × IQR = 168, so some texts would call it an extreme outlier. A systolic pressure of 184 mmHg is high but physiologically possible, so the next step is to check the source record, and the value is kept if it is confirmed. In practice these numbers are read from R rather than worked out: for the same eleven values quantile() reports Q1 = 108 and Q3 = 123, IQR() reports 15 and boxplot.stats(x)$out lists 184 as the only value beyond the fences, as part 2 of Activity 2.2, at the end of this section, shows. Textbooks differ slightly in how they place Q1 and Q3 for some sample sizes, so the quartiles that R prints are the ones to report.

The z-score method standardizes each observation as z = (x − mean) / SD. Values with |z| > 3 are commonly flagged as potential outliers.

Equation 2.2 gives the z-score as a formula.

Z-score
\[ \color{#0B7B6B}{z} = \frac{\color{#C2410C}{x} - \color{#6D28D9}{\bar{x}}}{\color{#1D4ED8}{s}} \]Eq 2.2
The z-score is the distance of a value from the mean, in units of the standard deviation. Values with an absolute z-score above 3 are commonly flagged.

This approach assumes an approximately normal distribution and can be misleading for heavily skewed data because the mean and SD are themselves influenced by outliers (the masking effect). Masking is the situation in which an extreme value hides itself, or hides other unusual values, from the rule meant to detect it. This happens because the z-score is calculated from the mean and SD of all the values, including the extreme ones. An extreme value pulls the mean towards itself and, because its squared distance from the mean is large, it inflates the SD. As a result every z-score becomes smaller, since each distance is now divided by a larger SD, and values that would otherwise exceed 3 can fall below it. The cut-off of 3 also rests on the normal distribution, so in a strongly right-skewed variable many genuine values in the long tail can be flagged while unusual low values are missed.

Worked Example 2.2 shows what the z-scores of two systolic blood pressure readings mean.

Worked Example 2.2: A z-Score

The standard deviation (SD) measures the typical distance of values from their mean; the next section explains how it is calculated. Suppose systolic blood pressure in a population has a mean of 120 mmHg and an SD of 15 mmHg. A reading of 180 mmHg has z = (180 − 120) / 15 = 60 / 15 = 4, so it lies four SDs above the mean. The absolute value |z|, the size of z ignoring its sign, is 4, which is above the usual cut-off of 3, so the reading is flagged. A reading of 135 mmHg has z = (135 − 120) / 15 = 1, one SD above the mean, which is ordinary. In a bell-shaped (normal) distribution about 99.7% of values lie within three SDs of the mean, which is why |z| > 3 is the conventional flag. In R, scale() converts every value of a variable to its z-score in one step, so in practice the z-scores are read from R output and the task is to decide which readings to investigate.

Visual inspection using boxplots, histograms, and scatterplots is often the most informative first step in outlier detection.

  • Boxplots display the median, IQR, and individual outliers beyond the whiskers
  • Histograms reveal gaps in the distribution that may indicate erroneous values
  • Scatterplots can reveal bivariate outliers that appear normal when each variable is examined individually

Visual methods should complement, not replace, quantitative rules. Always investigate outliers before deciding to exclude them.

The two worked examples show the main difference between the rules: the IQR rule rests on percentiles, which a few extreme values barely move, whereas the z-score rule rests on the mean and SD, which extreme values pull towards themselves.

3.2.2 Visual Checks

A boxplot draws the IQR rule directly. Figure 2.1 shows a boxplot of systolic blood pressure with the IQR fences marked and the points beyond them flagged.

Box plot of systolic blood pressure with the IQR outlier fences marked; five points fall beyond the fences and are flagged as outliers.
Figure 2.1. The box spans the interquartile range; the dashed lines are the lower and upper fences (Q1 − 1.5×IQR and Q3 + 1.5×IQR). Points beyond them, shown in orange, are flagged for investigation, not automatic deletion.

Two other kinds of problem are easier to see in a plot than in a table of numbers. A histogram (a bar chart of how many values fall in each range) shows gaps and separate clusters. When a decimal point is shifted during data entry, the shifted values form a small cluster at ten times the typical value, far from the main distribution, as in panel B of Figure 2.2.

Two histograms of simulated systolic blood pressure. Panel A shows one continuous cluster from about 75 to 170 mmHg. Panel B shows the same data with eight readings multiplied by ten, which appear as a small separate cluster between 1,200 and 1,450 mmHg, far to the right of the main bar.
Figure 2.2. Simulated data. A: a single continuous cluster, as expected for blood pressure. B: eight readings with a shifted decimal point (for example 1,280 entered for 128) form a separate cluster near 1,200 mmHg. The y-axis in panel B is cut at 25 so that the small cluster is visible.

A scatter plot shows each person as a point with two measurements. A bivariate outlier is a combination of values that is implausible even though each value is ordinary on its own. In the simulated example in Figure 2.3, a height of 191 cm and a weight of 46 kg each lie inside the boxplot whiskers, but together they imply a BMI of 12.6, which would be life-threatening. Only the scatter plot reveals the problem.

Panel A shows separate boxplots of height and weight; the highlighted person, at 191 cm and 46 kg, lies inside the whiskers of both. Panel B is a scatter plot of weight against height in which most points rise from lower left to upper right, and the highlighted point sits far below the cloud on the right.
Figure 2.3. Simulated data. A: each value of the highlighted person lies within the boxplot whiskers. B: the combination of 191 cm and 46 kg (BMI 12.6) sits far below the cloud of other people and is flagged for checking.

3.2.3 Outlier Rules in R

Both numerical rules can be applied to a whole variable in a few lines of R. quantile(x, probs = c(0.25, 0.75)) returns Q1 and Q3, IQR() returns their difference, boxplot.stats(x)$out lists the values beyond the fences that a boxplot would draw as points, scale() converts every value to its z-score, and which() returns the row numbers where a condition is TRUE, so that the IDs can be listed. Part 2 of Activity 2.2, at the end of this section, repeats the eleven-reading example, applies the IQR rule and z-scores to BMI in the raw course file, and counts the values that fall in the soft-limit zones of the codebook. In the course file the IQR rule flags five BMI values, one impossible (95) and four ordinary values for adults with obesity (30.3 to 31.6), whereas the z rule flags only the 95: that single value pulls the mean up and inflates the SD from 3.24 to 4.17, and without it three of the four readings have |z| > 3. One extreme error hides other unusual values from the z rule, which is the masking described in the Z-Score tab. For skewed data, or whenever one value may be an error, the IQR rule is therefore the safer of the two.

3.2.4 Keep, Transform or Exclude

Every rule and plot in this subsection flags values; none of them can show that a value is wrong. The rule that follows is one of the most often tested in this lesson.

Never Delete an Outlier Because It Is Extreme

Never delete an outlier simply because it is extreme. Investigate whether it represents a data error, a genuinely unusual observation, or a member of a different subpopulation. Document every decision to retain or remove an outlier in your audit trail.

Once a flagged value has been investigated, one of three decisions follows. The panel below sets out when each applies and how the decision is written down as a rule in the cleaning log.

Keep, Transform or Exclude: Deciding What to Do With a Flagged Value

Keep a value that is possible and has been confirmed, or at least not shown to be wrong, such as a systolic pressure of 184 mmHg. If several such values give the variable a long tail, the variable is summarised with the median and IQR, or analysed with a rank-based method, rather than altered.

Transform the whole variable, not a single value, when its values are real but strongly skewed, for example ln(x + 1) for a right-skewed count that contains zeros (see 3.3 below). A transformation changes the scale of every value; it does not repair an error, so it is never the answer for an impossible or miscoded value.

Exclude a value that is an error: one beyond a hard limit, a missing-value code read as a number, or a value that the source document shows to be wrong. It is corrected from the source if possible and otherwise set to NA, and the row is kept so that the person's other values remain available. A whole row is removed only when the person falls outside the study population, and that criterion is stated in the methods.

Writing the rule. A cleaning-log rule names the variable, the limit or condition that was broken, the action taken, how many values it affected, and who made the change and when. For example: "systolic_bp outside the hard limits of 80 to 220 mmHg is impossible for this survey; values checked against the source records; 2 unverifiable values (300 and 60) set to NA; rows retained; AB, 2026-09-30." Activity 2.2, at the end of this section, records the same information as a table.

The outlier rules and soft limits flag values for investigation and leave them unchanged unless the investigation shows an error. Of the three decisions, transforming is the one that changes a whole variable, and it is the subject of the next subsection.

3.3 Transformations and Derived Variables

3.3.1 Transforming Skewed Variables

When continuous variables are heavily skewed, transformations can make the distribution more symmetric, stabilize variance, and improve the validity of parametric methods. A transformation applies the same mathematical function, such as the logarithm, to every value of a variable. By this we mean that, with the log and square-root transformations, the values keep their order but the distances between them change: on a log scale, for example, the step from 1 to 10 is the same size as the step from 10 to 100, so large values are pulled in much more than small ones. This is why a long right tail shrinks. Stabilizing variance refers to a related problem. In many skewed variables, such as biomarker concentrations, groups with higher averages also have much larger spreads, and a transformation that compresses large values can make the spreads of the groups more alike, which many methods assume. Parametric methods, such as the t-test and linear regression, assume that the data (or the errors of a model) follow a particular distribution, usually the bell-shaped normal distribution, which is fully described by its mean and standard deviation. When that assumption is badly wrong, their standard errors and confidence intervals can be misleading. Rank-based (non-parametric) methods, described in the next section, make fewer assumptions.

Figure 2.4 shows the effect of a natural-log transform on a right-skewed variable: the long right tail is pulled in and the distribution becomes roughly symmetric. Table 2.4 lists three common transformations with their formulas and typical uses.

Two histograms: a right-skewed biomarker distribution on the left, and the same data after a natural-log transform on the right, now approximately symmetric.
Figure 2.4. A natural-log transform pulls a right-skewed variable toward symmetry, which often improves the fit of methods that assume approximate normality. Interpretation then shifts to the multiplicative scale.

Table 2.4. Three common transformations for skewed continuous variables, with their formulas and typical uses.

TransformationFormulaWhen to Use
Log (natural)ln(x) or ln(x + 1)Right-skewed data, multiplicative relationships (e.g., biomarker concentrations, income)
Square root√xCount data, moderately right-skewed distributions
Inverse (reciprocal)1/xStrongly right-skewed data; commonly used for rates and times

The log of zero does not exist (R returns -Inf, minus infinity), so a variable that contains zeros, such as the number of clinic visits, cannot be logged directly. The usual remedy is to add 1 to every value first, ln(x + 1), which turns 0 into ln(1) = 0 and keeps all values in the same order. The constant 1 is a convention; when most values are small, the choice of constant affects the results, and the report should state it. R has a dedicated function, log1p(), for ln(x + 1), and expm1() reverses it.

The reciprocal, 1/x, reverses the order of values: the largest value becomes the smallest. Analysts often use −1/x instead, so that larger original values still have larger transformed values and the direction of any association stays easy to read.

A transformed variable has to be interpreted on its new scale, and the results have to be returned to the original units before they are reported. The rule is general enough to commit to memory.

Interpret on the New Scale, Report on the Original Scale

When you transform a variable, all subsequent interpretations must account for the transformation. For example, a regression coefficient from a log-transformed outcome represents a multiplicative (rather than additive) change. By this we mean that the coefficient no longer says "the outcome is so many units higher"; it says "the outcome is so many times higher", or so many percent higher. This happens because equal steps on the log scale correspond to equal ratios on the original scale. Always back-transform results for reporting, so that readers see a value in the original units, such as mg/L, rather than a number on the log scale.

Worked Example 2.3 applies this rule to the simplest case, a mean calculated on the log scale; watch for which of the three numbers R prints is the one that goes in a report.

Worked Example 2.3: Back-Transforming a Log Mean

Nine C-reactive protein (CRP) values, a blood marker of inflammation, are 0.4, 0.8, 1.1, 1.5, 2.3, 3.0, 4.8, 9.5 and 21.0 mg/L. Part 5 of Activity 2.2 enters them in R. R prints an arithmetic mean of 4.93 mg/L, pulled upward by the single value of 21.0, and a median of 2.3 mg/L. On the log scale, mean(log(crp)) prints 0.906, and back-transforming with the exponential function, exp(mean(log(crp))), prints 2.48 mg/L. This back-transformed value is the geometric mean, a typical value on the original scale that is much less affected by the long right tail. A report would give the geometric mean of 2.48 mg/L, because the log-scale figure of 0.906 has no direct meaning for most readers. The same logic applies to regression coefficients for a log-transformed outcome, which are back-transformed into percentage differences.

The example shows that the geometric mean, unlike the arithmetic mean, stays close to the bulk of a right-skewed variable. Part 5 of Activity 2.2, at the end of this section, also demonstrates the other functions of this subsection: log() is the natural log and exp() reverses it, log1p() and expm1() are the matching pair for ln(x + 1), and the reciprocal reverses the order of 2, 5 and 10 (R prints 0.5, 0.2 and 0.1) while −1/x restores it.

3.3.2 Recoding and Derived Variables

A transformation changes the scale of an existing variable; recoding instead creates new variables from existing ones. Common examples include collapsing categories (e.g., combining “current smoker” and “former smoker” into “ever smoker”), categorizing continuous variables into clinically meaningful groups (e.g., BMI categories), and computing derived variables such as person-years of follow-up or age at onset.

In R, ifelse(condition, value_if_true, value_if_false) creates a new variable from a condition, and cut() turns a continuous variable into bands. In cut(), the breaks argument gives the cut-points, right = FALSE places each cut-point in the band above it (so a BMI of exactly 25.0 falls in the band 25 to 29.9), and labels names the bands. A cross-tabulation of the old and new variables, as in part 5 of Activity 2.2, is the standard check that a recode did what was intended. A derived variable is always stored as a new variable, so the original values are never overwritten and every result can be traced back to the raw data, and each new variable and its definition is added to the data dictionary.

3.3.3 String Cleaning and Duplicate Records

Free-text fields often contain inconsistencies: “Vancouver,” “vancouver,” “VANCOUVER,” and “Vancoouver” may all refer to the same city. Standardization strategies include converting to consistent case, trimming whitespace, applying regular expressions for pattern matching, and using lookup tables or fuzzy matching algorithms.

A regular expression is a short pattern that describes text. The pattern ^vanc means "text that starts with vanc" (the symbol ^ marks the start), and [0-9]+ means "one or more digits". The function grepl() asks whether a pattern occurs in each value, and gsub() replaces it. A lookup table is a two-column list that maps every known variant to its standard spelling. Fuzzy matching finds strings that differ by only a letter or two, which helps when the variants cannot all be listed in advance; its suggested matches should be checked by a person. The function duplicated() returns TRUE for the second and later copies of a value, so sum(duplicated(phaa$id)) counts repeated IDs.

Parts 5 and 6 of Activity 2.2, at the end of this section, apply these functions to five spellings of a city name, recode smoking status and BMI with ifelse() and cut(), and check the course file for repeated IDs. Lower case and trimws() merge three of the four spellings of Vancouver, and the pattern ^vanc catches the misspelling; a pattern this broad would also catch any other place name starting with "vanc", so the table is checked after every replacement. sum(duplicated(phaa$id)) is 0, so no ID appears twice; records entered twice under different IDs can be found with sum(duplicated(phaa[, names(phaa) != "id"])), which compares whole rows after setting the ID column aside and flags a record only when every other value matches. The recodes are stored in separate objects (city, smoker01, bmi_group) instead of new columns of phaa, so the cleaned file saved at the end of the lesson keeps the same columns as the course version of the dataset.

Every step so far, from range checks to recodes and text cleaning, changes or adds values, and each change has to be traceable. The next subsection sets out how these cleaning decisions are recorded.

3.4 Documenting the Cleaning Process

Every data cleaning decision should be recorded in an audit trail. This includes what was changed, why, by whom, and when. Reproducibility requires that another analyst could replicate the cleaning process from the original raw data using your documentation, an expectation echoed in the Statistical Methods in Psychology Journals guidelines (Wilkinson & the Task Force on Statistical Inference, 1999).

In practice the audit trail has two parts. The cleaning script records every change as code, with a comment giving the reason, and it is always run from the untouched raw file. A cleaning log, a small table with one row per changed value, records the date, the analyst, the ID, the variable, the old and new values and the reason, in a form that a reviewer can read without running any code. The cleaned data are then saved under a new file name so that the raw file is never overwritten; for the course data that file is phaa_survey_clean.csv, the survey after the range checks of this section, which Activity 2.3 in the next section reads from the course website.

An audit trail kept in a hypothetical cohort study of cardiovascular disease shows the two parts working together. The study collected blood pressure measurements on 2,400 participants, and during cleaning 15 systolic values exceeded 300 mmHg. The analyst traced 12 of these to a decimal-point shift (e.g., 1,200 instead of 120.0) and corrected them using the original paper forms. Three could not be verified and were set to missing with a note: “Original form illegible; value implausible.” All decisions were logged in a cleaning script with date stamps, so a reviewer could see every original value, every change and its reason, and could rebuild the cleaned file from the raw one. Correcting a value from the source document, as here, is good practice; what makes it acceptable is that the change is recorded and repeatable rather than typed over the raw file.

The range checks of 3.1 set five impossible values to missing, and they join the gaps that were already in the raw file. Deciding how to handle all of these missing values is the last step before the cleaned data are described, and it is the subject of the next subsection.

4.1 Choosing a Missing-Data Strategy

The choice of method for handling missing data depends on two things: the plausible missing-data mechanism, classified in the previous section as MCAR, MAR or MNAR, and the proportion of rows the analysis would lose. Practical guidance on implementing multiple imputation via chained equations, including how to specify the imputation model, how many imputations to draw, and how to diagnose problems, is given by White, Royston, & Wood (2011). The methods are compared in terms of five ideas, defined below, that recur whenever missing data are handled.

Five Terms for Comparing Missing-Data Methods

Complete-case analysis (also called listwise deletion) analyses only the rows that have no missing value on any variable used in the analysis.

Bias is a systematic difference between an estimate and the true value, so that repeating the study many times would give answers that are too high, or too low, on average.

Precision describes how much an estimate would vary from sample to sample; a precise estimate changes little when the study is repeated.

The standard error (SE) measures that sample-to-sample variation. For a mean it equals the standard deviation divided by the square root of the number of observations, so it shrinks as the sample grows.

A confidence interval (CI) is a range around an estimate, roughly the estimate plus or minus two standard errors for a 95% interval, that shows the values compatible with the data.

Missing data can damage both properties. Losing rows lowers precision (larger standard errors and wider confidence intervals), and missingness that is related to the values being studied, either directly (MNAR) or through observed variables linked to them (MAR), can bias a complete-case analysis.

Four methods are in common use, ranked here from the simplest to the one that handles uncertainty best: listwise deletion (complete-case analysis), pairwise deletion, single imputation and multiple imputation. Expand each panel below to see how each method works and where it goes wrong, paying particular attention to why single imputation makes standard errors too small and how multiple imputation avoids that.

Listwise Deletion (Complete-Case Analysis)

Excludes any observation with one or more missing values. Simple to implement, but can dramatically reduce sample size and introduces bias unless data are MCAR. The bias arises because the people who remain are no longer a fair picture of the whole sample when missingness is related to who they are. For example, if younger people skip the income question more often, the complete cases contain too few young people, and any summary that depends on age is distorted. In multivariable models with many variables, even low rates of missingness on individual variables can lead to large cumulative losses. This happens because a person is dropped if any one of the variables is missing, and the gaps usually fall on different people, so the losses add up across variables. The course data show this in part 7 of Activity 2.2: no single column is missing more than 4.4% of its values, yet only 687 of the 800 rows are complete.

Pairwise Deletion (Available-Case Analysis)

Uses all available data for each analysis. For example, when computing a correlation matrix, each pair of variables uses all cases with non-missing values for that pair. This preserves more data than listwise deletion, but can produce inconsistent results. For example, correlations computed on different subsets of people can form a set that could not all have come from one group of people (technically, a correlation matrix that is not positive semi-definite), which some later methods cannot use.

Single Imputation (Mean, Median, Mode)

Mean imputation replaces missing values with the variable mean. While simple, it underestimates variance and distorts relationships between variables. The variance is underestimated because every filled-in value sits exactly at the centre of the distribution, so the data look less spread out than real values would be. For example, if 100 missing BMI values in a hypothetical study are all replaced by the same mean value, the variable gains 100 values with no distance from the mean at all. Relationships are distorted for a similar reason: the filled-in values ignore everything else known about each person, so a person with high blood pressure receives the same average BMI as a person with low blood pressure, and the association between the two variables is weakened. Median imputation is more robust for skewed distributions. Mode imputation is used for categorical variables.

All single imputation methods treat the imputed value as if it were known with certainty, leading to underestimated standard errors and overly narrow confidence intervals. By this we mean that the analysis cannot tell an imputed value from a measured one. It therefore counts the imputed values as real information, believes it has more data than it actually has, and reports its estimates as more precise than they really are.

Multiple Imputation (Introduction)

Multiple imputation creates several (typically 5–20) plausible imputed datasets, analyzes each separately, and pools the results using Rubin’s rules. Each missing value is filled with a plausible value predicted from the person's other recorded variables, plus a random amount that reflects how uncertain that prediction is. Because the random part differs each time, the completed datasets differ slightly from one another in the values they fill in, while the observed values stay the same in all of them. This approach correctly accounts for the uncertainty due to missing data and produces valid inference under the MAR assumption.

Rubin’s rules average the estimates from the completed datasets and combine two sources of variation in the standard error: the ordinary variation within each dataset and the variation between datasets, which reflects how uncertain the imputed values are. By this we mean that if the completed datasets give very similar estimates, the missing values matter little and the pooled standard error is only slightly larger than it would be with complete data. If instead the estimates differ a lot from one dataset to the next, the missing values could plausibly have been quite different, and the pooled standard error grows to reflect that. This is the step that single imputation leaves out, and it is why multiple imputation does not overstate precision.

Multiple imputation is widely used for handling missing data in epidemiologic research when the proportion of missingness is non-trivial. It needs care: the imputation model should include the outcome and the variables that predict missingness (Sterne et al., 2009).

The panels set out the trade-off that drives the choice: the simpler methods either discard people or pretend to know values that were never recorded, while multiple imputation keeps everyone and carries the uncertainty about the missing values into the standard errors. Before any of them is applied, the analyst has to make sure that R recognises every missing value, then find out how much is missing and who is missing.

4.1.1 Missing-Data Codes Such as -9 and 999

Many data systems cannot leave a cell empty, so they store a code such as -9 (refused) or 999 (not asked) instead. R treats these codes as real numbers. A single age of 999 among the 800 ages in the course file would raise the mean by about 1.2 years, and in a small sample one code can move a mean by decades: in part 7 of Activity 2.2, at the end of this section, the mean of five ages that include the codes -9 and 999 is 224.4, while the mean of the three real ages is 44. The codebook lists the codes for each variable, and the analyst converts them to NA before any summary is calculated, for example with age_raw[age_raw %in% c(-9, 999)] <- NA. The course file uses empty cells only, so the activity uses a short made-up vector. The operator %in% asks, for each value on its left, whether it appears in the list on its right. The na.strings argument of read.csv(), for example na.strings = c("", "NA", "-9", "999"), converts codes while the file is read, but it applies to every column, so it is safe only when the code cannot be a real value in any column. A value of 999 could be a real number of minutes of activity per week, for example. Converting codes column by column with %in%, guided by the codebook, avoids that problem.

4.1.2 How Much Is Missing, and Who Is Missing?

Once every missing value is coded as NA, choosing a method starts with knowing how much is missing and whether the people with missing values differ from the rest. In R, colSums(is.na(phaa)) counts the missing values in each column, complete.cases() returns TRUE for each row with no missing value, na.omit() returns a copy of the data with every incomplete row removed, and tapply(y, group, mean) calculates the mean of y separately for each value of group. Part 7 of Activity 2.2, at the end of this section, uses these functions on the course file after the range checks. Eight columns then have gaps, the largest being income at 4.4%, yet only 687 of the 800 rows (85.9%) are complete: a complete-case analysis that used every column would lose 113 people (14%), because the gaps fall on different people, so na.omit() should be applied only to the columns an analysis needs, for example na.omit(phaa[, c("age", "systolic_bp")]). The 35 people without an income value are a little younger (42.6 versus 45.3 years), have slightly lower systolic BP (105.3 versus 108.4 mmHg) and smoke less often (8.6% versus 17.4%, which is 3 of the 35 people). Differences like these, if real, would argue against MCAR and suggest that missingness depends on observed variables (MAR). Because the course file is simulated, its true mechanism is known: the income values were deleted completely at random, so the differences arose by chance in a group of only 35 people. Comparisons of this kind can suggest a mechanism, and small groups can produce sizeable differences by chance alone, so they cannot establish one.

4.1.3 Reasoning About the Mechanism

The observed data can show that missingness is related to observed variables, which is evidence against MCAR. They cannot show whether missingness depends on the missing values themselves, because those values were never recorded. Whether MNAR is plausible is therefore judged from subject knowledge. Income is the standard example: refusal to answer income questions is often reported to be more common among people with high incomes, so an analyst should treat MNAR as a real possibility for income even when a comparison like the one in Activity 2.2 looks reassuring. By contrast, a blood pressure reading lost because a machine failed at random times is a case where MCAR is plausible.

Because the mechanism cannot be confirmed from the data, it is stated as an assumption in the methods section of a report. An example sentence is: "Income was missing for 35 of 800 participants (4.4%). We assumed that income was missing at random given age, gender, education and region, and handled it with multiple imputation; a sensitivity analysis that assumed non-responders had higher incomes than similar responders gave similar results." The sentence reports the amount missing, the assumed mechanism, the method used, and how the conclusion was checked.

A formal test of the MCAR assumption exists, but what it can show is limited, and over-reading it is a common mistake.

What Little’s MCAR Test Can and Cannot Show

Little’s test (Little, 1988) compares the means of the observed variables across groups of people with different patterns of missing values. A small p-value (a result that would be unlikely if the data were MCAR) is evidence that the data are not MCAR. A large p-value is weak reassurance: the test has little power (ability to detect a departure from MCAR) when few values are missing, and because it uses only observed values it cannot detect MNAR at all. The test is available in add-on R packages (for example as mcar_test() in the naniar package) and is not needed in this course; the group comparison in part 7 of Activity 2.2 examines the same question in a form that is easier to read.

Why the mechanism matters so much becomes clear when the same number of values goes missing in two different ways. Worked Example 2.4 compares a complete-case mean when the missing people are chosen at random with the same mean when the highest earners are the ones who do not answer; watch the direction of the error in each case.

Worked Example 2.4: Missing High Earners Bias the Mean

Ten people have annual incomes (in thousands of dollars) of 22, 28, 31, 35, 40, 46, 52, 60, 85 and 120, with a true mean of 51.9. Table 2.5 shows the complete-case mean of the eight people who remain when two of them do not answer.

Table 2.5. Complete-case means of the ten incomes when two people chosen at random, or the two highest earners, do not answer.

ScenarioPeople who do not answerComplete-case mean of the other eightError
Two people chosen at random (MCAR)31 and 5254.5+2.6
The two highest earners (MNAR)85 and 12039.25−12.65

When the missing people are chosen at random (MCAR), the error can go in either direction and averages out to zero over many repetitions; the only cost is a smaller sample. When the highest values are the ones that go missing (MNAR), every complete-case mean is pulled downward, and collecting more data of the same kind does not remove the error. Part 8 of Activity 2.2, at the end of this section, repeats the comparison with 10,000 simulated incomes. There set.seed() fixes the random numbers so that the results repeat exactly, rlnorm() draws right-skewed values, and runif(10000) < 0.30 is TRUE for a random 30% of people. R prints a true mean of 65.9 thousand dollars. With MCAR missingness the complete-case mean that R prints, 65.3, is close to the truth, and the standard error rises from 0.437 to 0.509 because fewer values remain: no bias, lower precision. With MNAR missingness, where the top quarter of earners leave the question blank far more often, the line mean(inc_mnar, na.rm = TRUE) prints 54.4, well below the true mean of 65.9, and no method that uses only the observed incomes can recover the missing information without further assumptions.

The worked example shows that the amount missing was the same in both scenarios; only the mechanism differed, and it alone decided whether the complete-case mean was biased. The amount missing still matters for the choice of method, as the next part explains.

4.1.4 How Much Missing Data Is Too Much?

No single threshold applies to every study. As a rough guide, many applied texts suggest that when fewer than about 5% of the rows needed for an analysis have missing values and MCAR is plausible, complete-case analysis is unlikely to change the conclusions much. When more is missing, when missingness is related to observed variables, or when MNAR is plausible, multiple imputation under a stated MAR assumption, together with a sensitivity analysis, is the safer choice. The proportion that matters is the share of rows lost from the specific analysis, which can be much larger than the share missing on any one variable, as the 687 complete rows in part 7 of Activity 2.2 show.

This guide also explains where complete-case analysis fits in this lesson. It is defensible when little is missing and MCAR is plausible, which is the situation in the scale activity in the next section, where na.omit() drops the 15 people (about 2%) who skipped one depression item before a factor analysis. It becomes hard to defend when many rows are lost or when the people with missing values differ from the rest.

When the guide points to multiple imputation, the work is done in R. The optional panel below goes further than the course requires: expand it to see a minimal multiple imputation with the mice package, with its output read line by line and compared with a complete-case estimate.

Going further: multiple imputation in R with the mice package

The mice package (multivariate imputation by chained equations) fills each missing value with a draw from a model built from the other variables, repeats this to create several completed datasets, analyses each dataset, and pools the results with Rubin’s rules. The reference code below imputes the 28 missing physical-activity values using age, gender, smoking status, BMI and social support, and estimates the mean weekly activity. It runs after the code of Activity 2.2, so phaa holds the cleaned data, and it needs the mice package (install.packages("mice"), once).

Reference code: a minimal multiple imputation with mice
library(mice)
mi_vars <- phaa[, c("age", "gender", "smoker", "bmi", "phys_act_min",
                    "social_support_score")]
mi_vars$gender <- factor(mi_vars$gender)
mi_vars$smoker <- factor(mi_vars$smoker)
imp <- mice(mi_vars, m = 5, seed = 2026, printFlag = FALSE)  # 5 completed datasets
fit <- with(imp, lm(phys_act_min ~ 1))   # lm(y ~ 1) estimates the mean of y
summary(pool(fit), conf.int = TRUE)      # pooled with Rubin's rules

# Complete-case estimate and its standard error, for comparison
mean(phaa$phys_act_min, na.rm = TRUE)
sd(phaa$phys_act_min, na.rm = TRUE) / sqrt(sum(!is.na(phaa$phys_act_min)))
Console output
> summary(pool(fit), conf.int = TRUE) term estimate std.error statistic df p.value 1 (Intercept) 175.732 2.938265 59.80808 748.7907 2.121669e-287 2.5 % 97.5 % conf.low conf.high 1 169.9638 181.5002 169.9638 181.5002 > mean(phaa$phys_act_min, na.rm = TRUE) [1] 176.0518 > sd(phaa$phys_act_min, na.rm = TRUE) / sqrt(sum(!is.na(phaa$phys_act_min))) [1] 2.964489
Code or outputWhat it means
mice(mi_vars, m = 5, seed = 2026, printFlag = FALSE)The function creates five completed datasets; seed makes the random draws repeatable and printFlag = FALSE hides the progress messages. When the package loads, R prints messages about functions it masks, which can be ignored.
with(imp, lm(phys_act_min ~ 1))The same analysis is run on each completed dataset. lm(y ~ 1) is a regression with no predictors, whose single coefficient (the intercept) is the mean of y.
estimate, std.error, 2.5 % and 97.5 %The pooled mean is 175.7 minutes per week with a standard error of 2.94 and a 95% confidence interval from 170.0 to 181.5. These are the numbers to report.
Complete-case mean and standard errorThe complete-case mean is 176.1 (SE 2.96). The two answers agree closely, as expected, because the values in this simulated file were removed completely at random. With MAR missingness in real data the two can differ noticeably, and the imputed estimate is the one that corrects the bias when the MAR assumption holds.

For a fuller demonstration on real survey data, open the narrated R walkthrough below, which applies the steps of this subsection, from counting and comparing missing values to multiple imputation, to the Canadian Social Connection Survey.

Learn to do this in R

Narrated R walkthrough: Missing Data and Multiple Imputation in R

The walkthrough counts the missing values in a survey dataset and lists their patterns, compares people with and without gaps to judge which mechanism is plausible, compares the people who have an outcome with the people who do not, fits a complete-case model, shows what mean and mode imputation do to the data, and then creates 40 imputed datasets with mice() and pools the results with Rubin’s rules.

Open the Missing Data and Multiple Imputation in R walkthrough

Every step of the cleaning workflow, from the first type check to the choice of a missing-data method, is now in place, and Activity 2.2 puts them together as one cleaning script. A cleaning script starts with a header that names the script, the raw file (which is never edited) and the file the script writes. Activity 2.2 practises the whole of this section in R on the raw course file: it checks the variable types, applies the outlier rules and the soft limits, replaces the values that break a hard limit, writes a cleaning log with one row for each value changed, transforms and recodes into new variables, cleans text, checks for duplicate IDs, and examines how much is missing and who is missing. Run the script along with the box, one part at a time, and compare each result with the notes.

RActivity 2.2: clean the course data and record every change

Text version of the activity: the code of the run-along script, its console output and notes on reading it.

phaa <- read.csv("https://heal-learn.ca/r-activities/data/phaa_survey.csv",   # course data, read from the website
                 stringsAsFactors = FALSE, na.strings = c("", "NA"))
summary(phaa$age)                  # before the conversion
phaa$age <- as.numeric(phaa$age)   # text such as "unknown" would become NA
phaa$bmi <- as.numeric(phaa$bmi)
summary(phaa$age)                  # after the conversion
Console output
Min. 1st Qu. Median Mean 3rd Qu. Max. -3.00 36.00 45.00 45.31 54.00 220.00 Min. 1st Qu. Median Mean 3rd Qu. Max. -3.00 36.00 45.00 45.31 54.00 220.00

The file is the raw course file phaa_survey.csv, the simulated survey of 800 adults introduced in Section 1, read straight from the course website; the raw file is never edited, so the cleaning can always be replayed from a known starting state. na.strings = c("", "NA") treats blank cells as missing. In this file every age cell holds a number, so read.csv() already stores age as a number and both summaries show 0 NAs: the conversion changes nothing here. Had a cell held text such as "unknown" or "30s", the column would have arrived as character and as.numeric() would have turned each such cell into NA, with a warning. The minimum of −3 and the maximum of 220 are the first sign of impossible ages.

x <- c(96, 102, 106, 110, 112, 115, 118, 121, 125, 131, 184)
quantile(x, probs = c(0.25, 0.75))   # Q1 and Q3
IQR(x)                               # Q3 minus Q1
boxplot.stats(x)$out                 # values beyond the fences
q <- quantile(phaa$bmi, probs = c(0.25, 0.75), na.rm = TRUE)
fences <- c(q[1] - 1.5 * IQR(phaa$bmi), q[2] + 1.5 * IQR(phaa$bmi))
fences
phaa[which(phaa$bmi < fences[1] | phaa$bmi > fences[2]), c("id", "bmi")]
z <- as.numeric(scale(phaa$bmi))     # z = (x - mean) / SD
phaa$id[which(abs(z) > 3)]           # IDs with |z| above 3
round(z[which(phaa$bmi == 31.6)], 2)
bmi_ok <- phaa$bmi[phaa$bmi < 60]    # without the impossible 95
round(c(sd_all = sd(phaa$bmi), sd_ok = sd(bmi_ok)), 2)
round((c(30.6, 30.8, 31.6) - mean(bmi_ok)) / sd(bmi_ok), 2)
sum(phaa$systolic_bp < 90 | phaa$systolic_bp > 180)   # soft limits
sum(phaa$bmi < 16 | phaa$bmi > 50)
Console output
25% 75% 108 123 [1] 15 [1] 184 25% 75% 11.45 30.25 id bmi 78 P0078 95.0 151 P0151 31.6 228 P0228 30.3 320 P0320 30.6 542 P0542 30.8 [1] "P0078" [1] 2.58 sd_all sd_ok 4.17 3.24 [1] 3.03 3.09 3.34 [1] 48 [1] 67

For the eleven readings of Worked Example 2.1, R prints Q1 = 108, Q3 = 123 and IQR = 15, and only 184 lies beyond the fences. For BMI the fences are 11.45 and 30.25, and five values are flagged. One (95.0, ID P0078) is impossible; the other four (30.3 to 31.6) are ordinary values for adults with obesity. The IQR rule finds what is unusual in this sample, and it cannot tell an error from a real value. Only P0078 has |z| > 3: the single value of 95 pulls the mean up and inflates the SD from 3.24 to 4.17, so the BMI of 31.6 has z = 2.58. Without the impossible value, the readings of 30.6, 30.8 and 31.6 have z = 3.03, 3.09 and 3.34. One extreme error hid three unusual values from the z rule, which is called masking, and it is why the IQR rule is safer for skewed data. The soft-limit counts flag values for a second look and change nothing: 48 systolic readings lie below 90 or above 180 mmHg (two of them, 60 and 300, are beyond the hard limits; the other 46 lie between 85 and 89 mmHg), and 67 BMI values lie below 16 or above 50 (one is the impossible 95; the other 66 lie between 15.0 and 15.9 and are kept).

phaa[which(phaa$age < 18 | phaa$age > 100), c("id", "age")]
phaa[which(phaa$systolic_bp < 80 | phaa$systolic_bp > 220), c("id", "systolic_bp")]
phaa[which(phaa$bmi < 13 | phaa$bmi > 60), c("id", "bmi")]
phaa$age[phaa$age < 18 | phaa$age > 100] <- NA
phaa$systolic_bp[phaa$systolic_bp < 80 | phaa$systolic_bp > 220] <- NA
phaa$bmi[phaa$bmi < 13 | phaa$bmi > 60] <- NA
summary(phaa$age)                  # 2 NAs, and every row is kept
phaa$education <- factor(phaa$education, levels = c("Less than high school",
  "High school", "Some college", "Bachelor's", "Graduate degree"), ordered = TRUE)
levels(phaa$education)             # lowest to highest
Console output
id age 33 P0033 -3 612 P0612 220 id systolic_bp 17 P0017 300 245 P0245 60 id bmi 78 P0078 95 Min. 1st Qu. Median Mean 3rd Qu. Max. NAs 18.00 36.00 45.00 45.15 54.00 87.00 2 [1] "Less than high school" "High school" "Some college" [4] "Bachelor's" "Graduate degree"

This is the editing stage of the screening, diagnosing and editing sequence. The first three lines list the values that break each hard limit in the codebook, with their IDs, before anything is changed: two ages (−3 and 220), two systolic pressures (300 and 60 mmHg) and one BMI (95). The next three lines replace them with NA. Replacing an impossible value with NA keeps the row, so the person's other 26 variables remain available, and the gap is then handled like any other missing value; deleting the row would discard valid data and could bias the results if errors are not random. summary(phaa$age) now shows 2 NAs while all 800 rows remain. Values in the soft-limit zones are left as they are. The last two lines make education an ordered factor, so that R treats "Less than high school" < "High school" < "Some college" < "Bachelor's" < "Graduate degree" as a direction; without the levels, R would sort the categories alphabetically, which makes no sense for a question such as "as education goes up, does blood pressure change?".

cleaning_log <- data.frame(date = "2026-09-30", analyst = "AB",
  id       = c("P0033", "P0612", "P0017", "P0245", "P0078"),
  variable = c("age", "age", "systolic_bp", "systolic_bp", "bmi"),
  old      = c(-3, 220, 300, 60, 95), new = NA,
  reason   = c("below 18: impossible", "above 100: impossible",
               "above 220: implausible", "below 80: implausible",
               "above 60: implausible"))
cleaning_log
Console output
date analyst id variable old new reason 1 2026-09-30 AB P0033 age -3 NA below 18: impossible 2 2026-09-30 AB P0612 age 220 NA above 100: impossible 3 2026-09-30 AB P0017 systolic_bp 300 NA above 220: implausible 4 2026-09-30 AB P0245 systolic_bp 60 NA below 80: implausible 5 2026-09-30 AB P0078 bmi 95 NA above 60: implausible

The cleaning log has one row for each value changed in part 3: the date, the analyst, the ID, the variable, the old and new values and the reason, in a form that a reviewer can read without running any code. Together with the cleaning script, which records every change as code and is always run from the untouched raw file, it forms the audit trail. write.csv(cleaning_log, "cleaning_log.csv", row.names = FALSE) saves the log beside the script, and the cleaned data are saved under a new file name so that the raw file is never overwritten.

crp <- c(0.4, 0.8, 1.1, 1.5, 2.3, 3.0, 4.8, 9.5, 21.0)   # CRP, mg/L
c(mean = mean(crp), median = median(crp))
mean(log(crp))                     # the mean on the log scale
exp(mean(log(crp)))                # back-transformed: geometric mean
visits <- c(0, 0, 1, 2, 5, 12)
log(visits)                        # log(0) is -Inf
log1p(visits)                      # log(x + 1), so 0 stays 0
expm1(log1p(visits))               # expm1() undoes log1p()
1 / c(2, 5, 10)                    # the reciprocal reverses the order
-1 / c(2, 5, 10)                   # a minus sign restores it
smoker01 <- ifelse(phaa$smoker == "Yes", 1, 0)   # 1 = smoker
table(phaa$smoker, smoker01)       # check every recode
bmi_group <- cut(phaa$bmi, breaks = c(0, 18.5, 25, 30, Inf), right = FALSE,
  labels = c("Under 18.5", "18.5 to 24.9", "25 to 29.9", "30 or more"))
table(bmi_group, useNA = "ifany")
Console output
mean median 4.933333 2.300000 [1] 0.9063658 [1] 2.47531 [1] -Inf -Inf 0.0000000 0.6931472 1.6094379 2.4849066 [1] 0.0000000 0.0000000 0.6931472 1.0986123 1.7917595 2.5649494 [1] 0 0 1 2 5 12 [1] 0.5 0.2 0.1 [1] -0.5 -0.2 -0.1 smoker01 0 1 No 664 0 Yes 0 136 bmi_group Under 18.5 18.5 to 24.9 25 to 29.9 30 or more <NA> 199 521 75 4 1

The arithmetic mean of the nine CRP values, 4.93 mg/L, is pulled upward by the 21.0, and the median is 2.3. The mean of the logs is 0.906, which has no direct meaning for most readers, so it is back-transformed with exp(), and line 58 prints the geometric mean, 2.48 mg/L, which goes in the report. log(0) is -Inf, so a count with zeros needs log1p(), ln(x + 1), which keeps 0 at 0 and keeps the order; expm1() reverses it. The reciprocal reverses the order of 2, 5 and 10, and −1/x restores it. The recodes are stored as new objects, so the original values are never overwritten: all 664 "No" values became 0 and all 136 "Yes" values became 1, and the BMI bands hold 199, 521, 75 and 4 people, with one person in no band because that BMI was set to NA in part 3 (useNA = "ifany" makes it visible).

city <- c("Vancouver", "vancouver ", "VANCOUVER", "Vancoouver", " Burnaby")
city <- trimws(tolower(city))      # remove stray spaces, lower case
city
city[grepl("^vanc", city)] <- "vancouver"   # starts with "vanc"
table(city)
sum(duplicated(phaa$id))           # IDs present more than once
Console output
[1] "vancouver" "vancouver" "vancouver" "vancoouver" "burnaby" city burnaby vancouver 1 4 [1] 0

Making the text lower case and removing stray spaces merges three of the four spellings of Vancouver; the misspelling "vancoouver" remains until the regular expression ^vanc (text that starts with vanc) catches it. A pattern this broad would also catch any other place name starting with "vanc", so the table is checked after every replacement. sum(duplicated(phaa$id)) is 0, so no ID appears twice. Records entered twice under different IDs can be found with sum(duplicated(phaa[, names(phaa) != "id"])), which compares whole rows after setting the ID column aside.

age_raw <- c(34, -9, 51, 999, 47)  # -9 = refused, 999 = not asked
mean(age_raw)                      # wrong: the codes count as ages
age_raw[age_raw %in% c(-9, 999)] <- NA
mean(age_raw, na.rm = TRUE)
n_miss <- colSums(is.na(phaa))
n_miss[n_miss > 0]                 # only the columns with gaps
round(100 * n_miss[n_miss > 0] / nrow(phaa), 1)
sum(complete.cases(phaa))          # rows with no gap in any column
nrow(na.omit(phaa))                # na.omit() keeps only those rows
inc_miss <- is.na(phaa$income)     # TRUE = income missing
table(inc_miss)
round(tapply(phaa$age, inc_miss, mean, na.rm = TRUE), 1)
round(tapply(phaa$systolic_bp, inc_miss, mean, na.rm = TRUE), 1)
round(prop.table(table(inc_miss, phaa$smoker), margin = 1) * 100, 1)
Console output
[1] 224.4 [1] 44 age income phys_act_min 2 35 28 social_support_score dep4 anx2 20 15 15 bmi systolic_bp 1 2 age income phys_act_min 0.2 4.4 3.5 social_support_score dep4 anx2 2.5 1.9 1.9 bmi systolic_bp 0.1 0.2 [1] 687 [1] 687 inc_miss FALSE TRUE 765 35 FALSE TRUE 45.3 42.6 FALSE TRUE 108.4 105.3 inc_miss No Yes FALSE 82.6 17.4 TRUE 91.4 8.6

A code such as −9 (refused) or 999 (not asked) is read as a real number: the mean of the five values is 224.4, while the mean of the three real ages is 44. Codes are converted column by column with %in%, guided by the codebook; na.strings in read.csv() applies to every column, so it is safe only when the code cannot be a real value anywhere (999 could be real minutes of activity). After the range checks, eight columns have gaps, the largest being income at 4.4%, yet only 687 of the 800 rows (85.9%) are complete, because the gaps fall on different people; na.omit() should be applied only to the columns an analysis needs. The 35 people without an income value are a little younger (42.6 against 45.3 years), have slightly lower systolic pressure (105.3 against 108.4 mmHg) and smoke less often (8.6% against 17.4%, 3 of the 35). Differences like these, if real, would argue against MCAR and suggest MAR. Because the file is simulated, the true mechanism is known: the income values were deleted completely at random, so these differences arose by chance in a small group. A comparison can suggest a mechanism but cannot establish one, and it can never show MNAR, because the missing values were never recorded.

set.seed(410)
income <- round(rlnorm(10000, meanlog = log(55), sdlog = 0.6))   # $1000s
mean(income)                       # the true mean
inc_mcar <- income
inc_mcar[runif(10000) < 0.30] <- NA          # MCAR: 30% for everyone
mean(inc_mcar, na.rm = TRUE)
p_miss <- ifelse(income > quantile(income, 0.75), 0.70, 0.15)
inc_mnar <- income
inc_mnar[runif(10000) < p_miss] <- NA        # MNAR: top earners 70%
mean(inc_mnar, na.rm = TRUE)
sd(income) / sqrt(10000)                     # SE of the mean, everyone
sd(inc_mcar, na.rm = TRUE) / sqrt(sum(!is.na(inc_mcar)))   # SE, MCAR
Console output
[1] 65.8979 [1] 65.34256 [1] 54.36383 [1] 0.437095 [1] 0.509361

The true mean of the 10,000 simulated incomes is 65.9 thousand dollars. With MCAR missingness the complete-case mean, 65.3, is close to the truth, and the standard error rises from 0.437 to 0.509 because fewer values remain: no bias, lower precision. With MNAR missingness, where the top quarter of earners leave the question blank far more often, the complete-case mean is 54.4, well below the true mean, and no method that uses only the observed incomes can recover the missing information without further assumptions; collecting more data of the same kind does not remove the error.

Files for this activity Data Code Answer key

R Reflect on what you just ran

Answer each question from your own output, as on the midterm: identify the problem, decide what to do with the values, and write down the decision.

1. The IQR rule (lines 19 to 22) flags five BMI values, but the z rule (lines 23 to 25) flags only one. Explain why the two rules disagree. For the five flagged values, state whether you would keep, transform or exclude them, and give your reasons. What happens to the 66 BMI values between 15.0 and 15.9 that lie in the soft-limit zone?

Model answerThe fences are 11.45 and 30.25, so the IQR rule flags the BMI of 95.0 (P0078) and four values from 30.3 to 31.6. The z rule flags only P0078 because that single value pulls the mean up and inflates the SD from 3.24 to 4.17, so the reading of 31.6 has z = 2.58; without the 95, the readings of 30.6, 30.8 and 31.6 have z = 3.03, 3.09 and 3.34, which is masking, and it is why the IQR rule is safer for skewed data. A BMI of 95 lies beyond the hard limit of 60 and cannot be correct, so it is excluded: it is corrected from the source record if possible and otherwise set to NA, with the row kept. The four values from 30.3 to 31.6 are possible values for adults with obesity, so they are kept; an outlier rule flags a value for investigation and cannot show that it is wrong. The 66 values between 15.0 and 15.9 lie between the soft limit (16) and the hard limit (13), so they are listed, checked against the source and kept unless they are shown to be wrong; transforming BMI would repair none of these values, because a transformation changes the scale of every value.

2. Lines 33 to 35 show impossible values in three variables. For each problem, state whether you would keep, transform or exclude the affected values, and write the rule you would record in the cleaning log. Why are the values set to NA rather than the rows deleted?

Model answerAge has two values beyond its hard limits of 18 to 100 years (−3 for P0033 and 220 for P0612), systolic blood pressure has two beyond 80 to 220 mmHg (300 for P0017 and 60 for P0245), and BMI has one beyond 13 to 60 (95 for P0078). All five are errors, so each is excluded: it is corrected from the source record if possible and otherwise set to NA, with the row kept; a transformation is not the answer, because it changes the scale and does not repair an impossible value. A rule names the variable, the limit or condition that was broken, the action taken, how many values it affected, and who made the change and when, for example "age outside the hard limits of 18 to 100 years is impossible for this survey of adults; values checked against the source; 2 unverifiable values (−3 and 220) set to NA; rows retained; AB, 2026-09-30". The rules for systolic_bp (hard limits 80 to 220 mmHg; 2 values, 300 and 60) and bmi (hard limits 13 to 60; 1 value, 95) follow the same pattern, and each value changed is one row of cleaning_log. Setting a value to NA keeps the person's other 26 variables in the analysis, records in the data that the value was flagged, and lets the gap be handled with the other missing values; deleting the row discards valid data and can bias the results if the errors are not random.

3. Run your own code: at the end of the script, type table(complete.cases(phaa[, c("age", "systolic_bp", "income")])) and run it. How many of the 800 people would a complete-case analysis of systolic blood pressure by income, adjusted for age, keep (TRUE), and how many would it lose (FALSE)? Using the comparison by income missingness (lines 89 to 93) and what you know about income, which missing-data mechanisms are plausible, and what is the consequence of analysing complete cases only under each? Which approach would you choose, and what would you state in the methods?

Model answerYour line shows 761 people with complete data on age, systolic blood pressure and income (TRUE) and 39 without (FALSE), so a complete-case analysis keeps 761 of the 800 and loses only 39, far fewer than are lost when every incomplete row is dropped (only 687 rows are complete, line 87). The 35 people without income are a little younger (42.6 against 45.3 years), have lower systolic pressure (105.3 against 108.4 mmHg) and smoke less (8.6% against 17.4%); if real, these differences mean that missingness depends on observed variables, which is missing at random (MAR), but with only 35 people they may be chance, and in this simulated file the incomes were in fact deleted completely at random (MCAR). Under MCAR a complete-case analysis remains unbiased and only loses precision (in the simulation, 65.3 against a true 65.9, with the standard error rising from 0.437 to 0.509); under MAR it can be biased, and multiple imputation that uses the variables that predict missingness (age and smoking) can recover valid estimates. The data cannot rule out missing not at random (MNAR), because people with high incomes are more likely to refuse the question; then the complete-case mean is biased (54.4 against 65.9 in the simulation), imputation under the MAR assumption does not correct it, and sensitivity analyses are required. With only 39 of the 800 people lost and MCAR plausible, a complete-case analysis is defensible here, and the methods should state the amount missing, the assumed mechanism and the method, with a sensitivity analysis (for example multiple imputation under MAR, or one that assumes non-responders have higher incomes) to check the conclusion.
Saved.

Activity 2.2 ends with a cleaned file, a cleaning log and a judgement about the missing values, which together are the products of this section. The section has set out the cleaning workflow, from systematic verification and outlier rules through transformations, recoding and string cleaning to the audit trail and the choice of a missing-data method. The key takeaways below summarise each step, the knowledge check that follows reviews them, and the next section turns the cleaned data into descriptive summaries.

Key Takeaways

  • 3.1 A first cleaning pass works through the variables systematically, using rules decided before you look at the data, rather than fixing whatever happens to catch your eye. Range checks compare each value with limits from the codebook: a hard limit marks values that cannot be correct (an age of 220 in a survey of adults, a systolic pressure of 300 mmHg), and a soft limit marks values that are possible but unusual (a systolic pressure of 85 mmHg). Type checks confirm that each column is stored as the codebook expects, consistency checks compare fields within a record (a discharge date must follow the admission date), and cross-validation compares sources, such as self-reported medication use against pharmacy records. When you read summary() output, start with the Min. and Max. lines and compare them with the documented valid range; a value that sits exactly at a limit is valid, so only values beyond it are problems. A value beyond a hard limit is corrected from the source if possible and otherwise set to NA, while a value between a soft and a hard limit is listed, checked against the source and kept unless it is shown to be wrong.
  • 3.2 Outlier rules flag values that sit far from the rest of the data; they cannot tell you whether a value is wrong. The IQR rule flags values below Q1 − 1.5 × IQR or above Q3 + 1.5 × IQR, and the z-score rule flags values with |z| > 3, where z = (x − mean) / SD. The IQR rule is safer for skewed data, because one extreme value pulls the mean, inflates the SD and can hide other unusual values from the z rule (masking), as the single BMI of 95 did in Activity 2.2. In R, quantile() and IQR() report the quartiles and the IQR, boxplot.stats(x)$out lists the values beyond the fences and scale() gives the z-scores: for the eleven readings of Worked Example 2.1, R reports Q1 = 108, Q3 = 123 and IQR = 15, which puts the fences at 85.5 and 145.5, so only the reading of 184 is flagged. Once a value is flagged, investigate it and then make one of three decisions: keep a possible value that has been confirmed or not shown to be wrong, transform the whole variable when its real values are strongly skewed, or exclude an error by correcting it from the source or setting it to NA while keeping the row. A transformation never repairs an impossible or miscoded value, and every decision is recorded with its reason.
  • 3.3 A transformation changes the scale of a whole variable to make a strongly right-skewed distribution more symmetric: the natural log, ln(x), suits right-skewed values such as biomarkers or income; ln(x + 1), log1p() in R, handles variables that contain zeros, because the log of 0 does not exist; the square root suits counts; and the reciprocal suits strongly skewed rates and times (use −1/x to keep the original order). After transforming, every interpretation is on the new scale, so back-transform for reporting: in Worked Example 2.3 the mean of the logged CRP values is 0.906, and exp(0.906) = 2.48 mg/L is the geometric mean that goes in the report. Derived variables, such as a 0/1 smoking indicator from ifelse() or BMI bands from cut(), are created as new variables so that the original values are never overwritten and every result can be traced back to the raw data. Check every recode with a cross-tabulation of the old and new variables, using useNA = "ifany" so that missing values show, and add each new variable and its definition to the data dictionary.
  • 3.4 The audit trail is what makes your cleaning reproducible: another analyst should be able to start from the untouched raw file and rebuild your cleaned dataset exactly. It has two parts. The cleaning script records every change as code, with a comment giving the reason, and is always run from the raw file, which is never edited by hand; the cleaning log is a small table with one row per changed value, recording the date, the analyst, the ID, the variable, the old and new values and the reason, as in Activity 2.2. When a question asks you to write a cleaning-log rule, include five things: the variable, the limit or condition that was broken, the action taken (corrected from the source, or set to NA with the row kept), how many values it affected, and who made the change and when. For example: "systolic_bp outside the hard limits of 80 to 220 mmHg; values checked against the source records; 2 unverifiable values set to NA; rows retained; AB, 2026-09-30."
  • 4.1 Choosing how to handle missing data starts with two facts: the plausible mechanism and how many rows the analysis would lose. Complete-case analysis (listwise deletion) is simple and defensible when little is missing (as a rough guide, under about 5% of the rows the analysis needs) and MCAR is plausible, but it can be biased under MAR or MNAR. Single imputation with the mean, median or mode fills each gap with one value and then treats that value as if it were known, so standard errors are too small and confidence intervals too narrow. Multiple imputation creates several plausible completed datasets (typically 5 to 20), analyses each one and pools the results with Rubin's rules, which gives valid estimates under MAR when the imputation model includes the outcome and the variables that predict missingness. Before you choose, count the complete rows for the variables your analysis actually uses with complete.cases() or na.omit(), because small gaps add up across variables: in the course file no variable is missing more than 4.4% of its values, yet only 687 of the 800 rows are complete. Then state the assumed mechanism and the method in your methods section, and check the conclusion with a sensitivity analysis.
Knowledge check: this section

1. In a birth registry extract of 1,500 births, the codebook gives hard limits of 20 to 44 weeks for gestational age (ga_wks) and soft limits of below 24 or above 42 weeks. The first cleaning pass prints the output below. Which plan follows the cleaning rules taught in this lesson?

> summary(births$ga_wks)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  22.0    37.0    39.0    39.3    40.0   390.0
> sum(births$ga_wks > 44)
[1] 2
> sum(births$ga_wks < 24)
[1] 3
Values between the soft and hard limits are possible, for example a very preterm birth, so they are checked and kept unless shown wrong. Only values beyond a hard limit are treated as errors.
Values beyond a hard limit cannot be correct, so they are corrected from the source or set to NA with the rows kept. Values beyond only a soft limit are possible, so they are checked and kept unless shown wrong.
A transformation changes the scale of every value but does not repair an impossible one. The value beyond the hard limit is an error to correct or set to NA.
Deleting rows discards each record's valid data on every other variable, and some of the flagged values are possible. Errors are set to NA with rows kept, and possible values are checked.
The answer is (b). The information needed is the two kinds of limit in the codebook and the counts beyond each: two values exceed the hard limit of 44 weeks, and three fall between the hard limit (20) and the soft limit (24). A value beyond a hard limit cannot be correct, so it is corrected from the source or set to NA with the row kept; a value between a soft and a hard limit is possible (a very preterm birth), so it is checked and kept unless it is shown to be wrong. Option (a) treats possible values as errors, and option (c) fails because a transformation changes the scale of every value but does not repair an impossible one. (Section 2, 3.1 Systematic Verification: Range, Type and Consistency Checks, Range Checks panel; Objective 3.1)

2. Lengths of hospital stay (days) for 11 patients are checked in R with the IQR rule and with z-scores. Which statement reads the output below correctly?

> los <- c(1, 2, 2, 3, 3, 4, 4, 5, 6, 7, 19)
> q <- quantile(los, probs = c(0.25, 0.75))
> q
25% 75%
2.5 5.5
> IQR(los)
[1] 3
> c(lower = q[[1]] - 1.5 * IQR(los), upper = q[[2]] + 1.5 * IQR(los))
lower upper
   -2    10
> boxplot.stats(los)$out
[1] 19
> round(as.numeric(scale(los))[11], 2)   # z-score of the 19-day stay
[1] 2.81
Lying beyond a fence flags a value for investigation; it does not prove an error. A long hospital stay is possible, so it is checked against the record and kept if confirmed.
The z-score rule suffers from masking: the extreme value inflates the mean and SD, which keeps its own z-score below 3. That is why the IQR rule suits skewed data, and the value still needs checking.
A flag is a reason to investigate, not to delete. Removing the row would discard a possibly genuine stay and the patient's other data.
The output gives the quartiles, the IQR and fences, and only one stay lies beyond them. A flagged value is checked against the record and kept if confirmed, because extreme is not the same as wrong.
The answer is (d). R prints the quartiles (2.5 and 5.5), the IQR (3), the fences (Q1 − 1.5 × IQR = −2 and Q3 + 1.5 × IQR = 10) and, through boxplot.stats(), the one value beyond them, 19. A flag is a reason to investigate, not proof of an error: a 19-day stay is long but possible, so it is checked against the record and kept if it is confirmed, never set to NA (a) or deleted (c) because it is extreme. Option (b) shows the masking effect: the 19-day stay itself inflates the mean and SD, so its z-score stays below 3 although it lies far from the other values, which is why the IQR rule suits skewed data. (Section 2, 3.2.3 Outlier Rules in R and 3.2.4 Keep, Transform or Exclude; Objective 3.2)

3. A survey of 2,400 adults will estimate mean BMI and its association with household income. BMI is missing for 22% of respondents. Missingness is higher among older respondents and among women, and age and sex are recorded for everyone. Which approach to the missing BMI values suits this situation best?

With a large share missing and missingness explained by recorded age and sex, MAR is plausible. Multiple imputation using those predictors and the analysis variables, pooled with Rubin's rules plus a sensitivity analysis, suits this case.
Imputed values are not real, but multiple imputation reflects their uncertainty when pooled. A complete-case analysis would discard over a fifth of the sample and can be biased under MAR.
Mean imputation keeps everyone but treats a filled-in value as known, so variance is understated, intervals are too narrow, and the association with income is distorted. The precision is false.
Pairwise deletion preserves data but is not a remedy for bias, and its results describe different subsets of people that may not fit together coherently. It does not use age and sex to address MAR.
The answer is (a). Two facts in the stem decide the method: the amount missing (22%, far more than a trivial share) and the mechanism (missingness depends on recorded variables, age and sex, so MAR is plausible). Under MAR a complete-case analysis (b) discards more than a fifth of the sample and can be biased, whereas multiple imputation that uses the predictors of missingness gives valid estimates and, through Rubin's rules, standard errors that include the uncertainty of the imputed values. Option (c) is tempting because no one is dropped, but a single imputed mean is treated as if it were known, so the variance is understated, confidence intervals are too narrow, and the association with income is distorted. (Section 2, 4.1 Choosing a Missing-Data Strategy and 4.1.4 How Much Missing Data Is Too Much?; Objective 4.1)

4. Minutes of vigorous activity per day are 0 for 35% of respondents and have a long right tail (skew 2.8). To reduce the skew, a colleague proposes the line dat$vig_min <- log(dat$vig_min). Which response is best?

The log of 0 does not exist, so R returns -Inf, and the inactive respondents become unusable. The line also overwrites the original column, losing the audit trail to the raw values.
Replacing zeros with the mean invents activity for people who reported none. The usual remedy for zeros is to add a constant before logging.
Adding 1 before logging handles the zeros and keeps the order of values. Writing to a new column and recording the transformation keeps the audit trail needed for interpretation and back-transformation.
The reciprocal does not solve the problem, because 1/0 is also undefined. The lesson's remedy for zeros with a log transformation is to add a constant first.
The answer is (c). The information needed is that the variable contains zeros and that the proposed line overwrites the original column. Because ln(0) does not exist (R returns -Inf), the usual remedy is ln(x + 1), and writing the result to a new column keeps the audit trail back to the raw values, on which every later interpretation and back-transformation relies. Option (a) would make the 35% of inactive people unusable and so lose them from the analysis, (b) invents activity for people who reported none, and (d) fails because 1/0 is also undefined. (Section 2, 3.3.1 Transforming Skewed Variables and 3.3.2 Recoding and Derived Variables; Objective 3.3)

5. During the first cleaning pass, sum(dat$sleep_hrs > 24) returns 4 for hours of sleep per night, whose codebook hard limits are 0 to 24 hours. The source forms do not show the true values. Which entry in the cleaning log records the decision correctly?

A transformation never repairs an impossible value; the values above 24 would stay in the data on a new scale. Impossible values are excluded by setting them to NA.
This entry names the variable, the broken limit, the action, the number affected, the script, and who made the change and when, so another analyst could repeat it. Setting unverifiable impossible values to NA while keeping rows is the correct action.
The count is recorded, but removing whole rows discards valid data on every other variable. The entry also omits the rule, the source check, and who made the change and when.
This records neither the rule nor the count, and describes a hand edit in a spreadsheet, which leaves no audit trail. Changes are made in a script and logged in full.
The answer is (b). A cleaning-log rule names the variable, the limit or condition that was broken, the action taken, how many values it affected, and who made the change and when, so that another analyst could repeat it. More than 24 hours of sleep in one night is impossible and the source cannot supply the true value, so the values are excluded by setting them to NA, while the rows are kept for the participants' other answers. Option (c) is tempting because it states the number affected, but removing whole rows discards valid data on every other variable; option (a) uses a transformation, which never repairs an impossible value; and option (d) records neither the rule nor the count, and describes a hand edit. (Section 2, 3.2.4 Keep, Transform or Exclude and 3.4 Documenting the Cleaning Process; Objectives 3.2 and 3.4)

6. A smoking-cessation programme sends the intake file for its 640 adult clients. The data dictionary gives hard limits of 0 to 80 for the number of cigarettes smoked per day (cig_day) and states that the age at which a client started smoking (age_start) cannot be greater than the client's current age (age). The first cleaning pass prints the output below. Which plan follows the cleaning rules taught in this section?

> summary(intake$cig_day)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
   0.00    8.00   15.00   16.42   20.00  200.00
> q <- quantile(intake$cig_day, probs = c(0.25, 0.75))
> c(lower = q[[1]] - 1.5 * IQR(intake$cig_day), upper = q[[2]] + 1.5 * IQR(intake$cig_day))
lower upper
  -10    38
> sort(intake$cig_day[intake$cig_day > 38])   # values beyond the upper fence
[1]  40  40  40  50  60 200 200
> sum(intake$age_start > intake$age)
[1] 5
A transformation changes the scale of every value but cannot repair an impossible one, so the values of 200 would still be wrong. The inconsistent ages are also left unaddressed.
The IQR rule flags values for investigation but cannot prove them wrong, and heavy smoking is possible. The inconsistent ages fail a consistency check even though each value is in range on its own.
Each problem is handled by its rule: values beyond the hard limit are corrected or set to NA, values flagged only by the fence are checked and kept unless wrong, and failed consistency checks are resolved at the source, with each rule logged.
Deleting rows discards valid data on every other variable, and several of the flagged values are possible. Problems are fixed value by value, with rows kept.
The answer is (c). Three rules from this section settle each problem. A value beyond a hard limit cannot be correct, so the two values of 200 are corrected from the source or set to NA with the rows kept. The IQR fences of −10 and 38 flag values for investigation but cannot show that a value is wrong, and 40 to 60 cigarettes a day is possible for a heavy smoker, so those five values are checked and kept unless they are shown to be wrong; option (b) treats a flag as proof of an error. A starting age greater than the current age is impossible even though each value lies in a plausible range, and only a consistency check, which compares fields within a record, finds it. Option (a) fails because a transformation changes the scale of every value but does not repair an impossible one (the two values of 200 would still be the largest), and option (d) discards valid data on every other variable. (Section 2, 3.1 Systematic Verification: Range, Type and Consistency Checks, 3.2.4 Keep, Transform or Exclude and 3.4 Documenting the Cleaning Process; Objectives 3.1, 3.2, 3.3 and 3.4)

● Complete the knowledge check to continue.

Section 3

Descriptive Analyses

⏱ Estimated time: 55 minutes
Section 3 of 3

Descriptive Analyses

Measures of central tendency, spread, and shape; visualisations; scale scores; Table 1; normality assessment.

Central tendency

Three measures; one rule for choosing

Arithmetic mean
\[ \color{#0B7B6B}{\bar{x}} = \frac{1}{n} \sum_{i=1}^{n} \color{#1D4ED8}{x_i} \]
x̄ mean x_i each observed value n number of observations

Mean

Uses all data; sensitive to skew and outliers. Report with SD for normal data.

Median

Resistant to extremes. Report with IQR for skewed data or when outliers are present.

Mode is appropriate for nominal categorical data where neither mean nor median is meaningful.

Spread and shape

How far and how symmetric

Sample standard deviation
\[ \color{#0B7B6B}{s} = \sqrt{\frac{\sum_{i=1}^{n}(\color{#1D4ED8}{x_i} - \color{#6D28D9}{\bar{x}})^2}{\color{#C2410C}{n}-1}} \quad \color{#047857}{\text{IQR}} = Q_3 - Q_1 \]
s standard deviation x_i each value x̄ mean n sample size IQR interquartile range
Right-skewed median mean Symmetric (normal) mean = median
Categorical data

Cross-tabulations and the odds ratio

Odds ratio from a 2×2 table
\[ \color{#0B7B6B}{OR} = \frac{\color{#C2410C}{a}\,\color{#6D28D9}{d}}{\color{#1D4ED8}{b}\,\color{#BE185D}{c}} \]
OR odds ratio a exposed cases d unexposed non-cases b exposed non-cases c unexposed cases
CasesControls
Exposed45 (a)30 (b)
Unexposed15 (c)60 (d)

OR = (45 × 60) / (30 × 15) = 6.0. A strong association, visible from the descriptive table alone.

Visualisation

Match the plot to the question

Histograms

Distribution of a continuous variable: shape, modality, gaps, outliers.

Boxplots

Median + IQR + outlier points. Best for comparing groups side by side.

Bar charts

Frequencies for categorical variables. Not for continuous data.

Scatter / line

Continuous relationships and trends over time.

Building scales

From questionnaire items to a score

Internal consistency

Cronbach's α: ≥ 0.70 acceptable, ≥ 0.80 good. The seven depression items in the course data reach α = 0.904.

Dimensionality

Exploratory factor analysis + parallel analysis. Loadings ≥ ~0.40 contribute meaningfully; depression and anxiety items load on separate factors.

Both checks pass → combine the items into a derived score (dep_score) and carry it into later lessons.

Table 1

The publication-ready summary of your cohort

CharacteristicExposedUnexposedp
Age, mean ± SD52.3 ± 11.449.8 ± 12.10.02
Female, n (%)110 (55%)162 (54%)0.82
BMI, median (IQR)27.1 (24.0–31.5)25.8 (23.2–29.4)0.004

Normal variables: mean ± SD. Skewed variables: median (IQR). Categorical: n (%). Missing-data counts belong in the table too.

Normality

Is it close enough to normal?

Visual methods

Q-Q plot: points along the diagonal suggest normality; systematic departures signal skew or heavy tails. Histogram with normal overlay for a quick read.

Formal tests

Shapiro-Wilk (preferred for n < 5,000); Kolmogorov-Smirnov as an alternative. In very large samples they flag trivial departures, so pair them with a plot.

Approximately normal → t-tests, ANOVA, Pearson. Otherwise → transform, or use rank-based tests (Wilcoxon rank-sum, Kruskal-Wallis, Spearman).

Carry forward

What this lesson set up

  • Choose central tendency and spread based on variable type and distributional shape.
  • A well-stratified Table 1 is often the most informative single output of an analysis.
  • Normality assessment should combine visual tools (Q-Q plot) with formal tests; neither alone is sufficient.
  • The clean, described dataset is the foundation on which every regression in later lessons rests.

Overview

The previous section produced a clean, well-documented dataset. This section turns that dataset into the descriptive summaries that any analysis report opens with. It starts with the measures of centre, spread and shape for a single variable and how to choose among them from R output, moves on to frequency tables, cross-tabulations and the odds ratio for categorical variables, and then builds a “Table 1” of participant characteristics stratified by exposure. It closes with the plots that suit each kind of variable and question, and with the checks that decide whether a variable is close enough to normal for parametric methods. Doing this carefully is what lets readers understand what your data look like before they trust your inferential models.

Learning Objectives

  • 5.2 Choose the appropriate measure of central tendency, spread, and shape for each variable type and distribution, justify the choice from summary() and describe() output, and describe or sketch the shape that output implies.
  • 6.1 Build frequency distributions and cross-tabulations to summarise categorical data, and obtain in R and interpret an odds ratio from a 2×2 table.
  • 6.2 Construct a publication-ready “Table 1” of participant characteristics, including stratified comparisons.
  • 7.1 Select visualisations that match the variable type and the question you are asking.
  • 7.2 Assess normality and decide when a transformation or non-parametric approach is warranted.

5.2 Choosing Measures of Centre, Spread and Shape

5.2.1 Measures of Central Tendency

Measures of central tendency summarize the “typical” value in a distribution. Three are in common use, the mean, the median and the mode, and the choice among them depends on the variable type and the shape of its distribution. Open each tab below to see how each measure is defined (Equation 2.3 gives the mean as a formula) and when each should be used.

The arithmetic mean (x̄) is the sum of all values divided by the number of observations. It uses all data points and is the basis of many parametric statistical methods. Parametric methods are those that assume the data follow a particular distribution, usually the bell-shaped normal distribution, which is fully described by its mean and standard deviation.

Equation 2.3 expresses the mean as a formula.

Arithmetic mean
\[ \color{#0B7B6B}{\bar{x}} = \frac{1}{\color{#C2410C}{n}} \sum_{i=1}^{\color{#C2410C}{n}} \color{#1D4ED8}{x_i} \]Eq 2.3
The mean is the sum of all observed values divided by the number of observations.

When to use: Symmetric or approximately normal distributions. Avoid for heavily skewed data or when outliers are present, as the mean is pulled toward extreme values.

The median is the middle value when observations are ordered from smallest to largest. For an even number of observations, it is the average of the two middle values.

When to use: Skewed distributions, ordinal data, or when outliers are present. The median is robust to extreme values; unlike the mean, adding or removing an outlier has minimal effect on the median.

The mode is the most frequently occurring value. A distribution may be unimodal, bimodal, or multimodal.

When to use: Categorical (nominal) data, where neither the mean nor the median is meaningful. It is also useful for identifying common response values in continuous data (e.g., digit preference). Digit preference is the tendency of people who record measurements to favour certain final digits, usually 0 and 5. For example, if blood pressure is read by eye from a gauge, readings of 120 or 130 mmHg may appear far more often than 121 or 127, so the mode lands on a round number. Spotting a mode of this kind warns the analyst that the values were rounded during measurement rather than recorded exactly.

The tabs make one contrast central: the mean uses every value and so is pulled toward extreme ones, whereas the median depends only on the middle of the ordered values. Worked Example 2.5 shows that contrast with a single extreme income; watch how far each measure moves.

Worked Example 2.5: An Extreme Income Moves the Mean

Five people earn 30, 35, 40, 45 and 50 thousand dollars a year. For these five values R's mean() and median() both print 40 (Worked code 2.10, below). A sixth person who earns 500 thousand dollars joins the group. The mean that R prints becomes 116.7, far above what five of the six people earn, while the median moves only to 42.5, halfway between 40 and 45, the two middle values of the six. The median still describes a typical member of the group, whereas the mean now lies far from five of the six incomes.

One extreme value moved the mean by more than 75 thousand dollars and the median by only 2.5 thousand, which is why skewed variables such as income are summarised with the median. Measures of central tendency describe the typical value; the next part describes how widely values spread around it.

5.2.2 Measures of Spread

Spread (or dispersion) describes how much variability exists in the data around the central value. Table 2.6 defines four common measures of spread and sets out their properties; note which of them, like the median, resist extreme values.

Table 2.6. Four measures of spread, with their formulas or definitions and their properties.

MeasureFormula / DefinitionProperties
RangeMaximum − MinimumSensitive to outliers; uses only 2 data points
Variance (s²)Σ(xi − x̄)² / (n − 1)Average squared deviation; units are squared
Standard Deviation (s)√s²Same units as original variable; most commonly reported
IQRQ3 − Q1Robust to outliers; covers middle 50% of data

The variance divides by n − 1 instead of n. The deviations are measured from the sample mean, which sits in the middle of the sample by construction, so they are slightly smaller on average than deviations from the true population mean would be. Dividing by n − 1, a slightly smaller number, corrects for this. The standard deviation is the square root of the variance, which brings the value back from squared units (squared mmHg, for instance) to the original units so that it can be reported beside the mean.

Worked Example 2.6 opens up the formula for the variance in Table 2.6 to show what the standard deviation reported by sd() measures, using five systolic readings; Table 2.7 sets out the deviation and squared deviation of each reading, as R prints them in Worked code 2.10.

Worked Example 2.6: What a Standard Deviation Measures

Five systolic readings are 110, 116, 120, 124 and 130 mmHg, with a mean of 120 mmHg.

Table 2.7. Deviations from the mean and squared deviations for the five systolic blood pressure readings.

Value xDeviation x − 120Squared deviation
110−10100
116−416
12000
124416
13010100
Sum0232

The deviations always sum to zero, which is why they are squared before being averaged. Dividing the sum of squared deviations by n − 1 gives the variance, which R prints as 58 squared mmHg, and its square root, the standard deviation, which sd() prints as 7.62 mmHg: a typical reading lies about 7.6 mmHg from the mean. R reports these values directly, so the skill to practise is reading the output and choosing what to report.

The worked example shows that the SD is a typical distance from the mean, expressed in the units of the data. Worked code 2.10 shows that mean(), median() and sd() give the numbers of both worked examples, and it builds the standard deviation step by step so that each line of output matches a column of Table 2.7.

Worked code 2.10: mean, median and standard deviation
income5 <- c(30, 35, 40, 45, 50)        # five incomes, $1000s per year
mean(income5)
median(income5)
income6 <- c(income5, 500)              # add one very high earner
mean(income6)
median(income6)

# Standard deviation, step by step
sbp <- c(110, 116, 120, 124, 130)
dev <- sbp - mean(sbp)                  # distance of each value from the mean
dev
dev^2                                   # squared distances
sum(dev^2) / (length(sbp) - 1)          # the variance
sqrt(sum(dev^2) / (length(sbp) - 1))    # the standard deviation
sd(sbp)                                 # R's built-in function gives the same
Console output
> income5 <- c(30, 35, 40, 45, 50) > mean(income5) [1] 40 > median(income5) [1] 40 > income6 <- c(income5, 500) > mean(income6) [1] 116.6667 > median(income6) [1] 42.5 > sbp <- c(110, 116, 120, 124, 130) > dev <- sbp - mean(sbp) > dev [1] -10 -4 0 4 10 > dev^2 [1] 100 16 0 16 100 > sum(dev^2) / (length(sbp) - 1) [1] 58 > sqrt(sum(dev^2) / (length(sbp) - 1)) [1] 7.615773 > sd(sbp) [1] 7.615773

Centre and spread are reported as a pair, and the pair is chosen together. The convention below sets out which pair to report.

Reporting Convention for Centre and Spread

For normally distributed variables, report mean ± SD. For skewed distributions, report median (IQR) or median (Q1, Q3). This convention is standard in epidemiologic publications and should be followed in “Table 1”.

The choice between these conventions depends on the shape of the distribution, which the next part describes.

5.2.3 Measures of Shape

Beyond central tendency and spread, the shape of a distribution provides critical information for choosing analytical methods. Two numbers describe it: skewness, which measures asymmetry, and kurtosis, which measures how heavy the tails are. Expand each panel below for the definition of each measure, how to read its sign, and the rule of thumb for a substantial departure; note in the second panel that describe() reports excess kurtosis, which is 0 for a normal distribution.

Skewness

Skewness measures the asymmetry of a distribution. By this we mean how much more the values stretch out on one side of the centre than on the other. A skewness of 0 indicates perfect symmetry: the left half of the histogram is a mirror image of the right half. The sign of the skewness names the side of the longer tail, and its size grows as that tail becomes longer and heavier. Skewness also explains why the mean and median separate, because the few values in a long tail pull the mean toward that tail while the median, which depends only on the middle value, stays with the bulk of the data.

  • Positive (right) skew: Long right tail; mean > median. Common in variables like income, hospital length of stay, and biomarker concentrations.
  • Negative (left) skew: Long left tail; mean < median. Less common but seen in variables like age at death in developed countries.

As a rule of thumb, |skewness| > 1 indicates substantial skew, though this depends on sample size.

Kurtosis

Kurtosis describes the “tailedness” of a distribution relative to a normal distribution (which has a kurtosis of 3, or excess kurtosis of 0). By this we mean how often values far from the centre occur, compared with how often they would occur in a normal distribution with the same mean and SD. Excess kurtosis is simply kurtosis minus 3, so that a normal distribution scores 0 and the sign shows the direction of the departure. Unlike skewness, kurtosis says nothing about which side the unusual values lie on; a distribution can be symmetric, or nearly so, and still have heavy tails on both sides, as panel D of Figure 2.5 shows.

  • Leptokurtic (excess kurtosis > 0): Heavier tails and a sharper peak; more extreme values than expected under normality.
  • Platykurtic (excess kurtosis < 0): Lighter tails and a flatter peak; fewer extreme values than expected.

High kurtosis can indicate the presence of outliers or a mixture of subpopulations. Outliers raise kurtosis because they add values far out in the tails. A mixture can do the same: if most of a sample comes from one group with values close together and a smaller group has very different values, the combined histogram has a narrow central peak with a few values spread far from it. For this reason a high kurtosis is a prompt to look at the histogram and the flagged values, not a conclusion in itself.

The describe() function in the psych package, used in Worked code 2.11, reports excess kurtosis. A value near 0 therefore means tails like those of a normal distribution, a positive value means heavier tails, and a negative value means lighter tails.

The two measures are easiest to understand by seeing them beside the shapes they describe. Figure 2.5 shows four simulated distributions with their skew and excess kurtosis, and marks the mean and median of each; compare where the two lines sit in each panel.

Four histograms of simulated data. A, symmetric, has skew 0.12 and excess kurtosis 0.08, with the mean and median lines on top of each other. B, right-skewed, has skew 1.21 and excess kurtosis 1.85, with a long right tail and the mean to the right of the median. C, left-skewed, has skew -1.60 and excess kurtosis 4.28, with a long left tail and the mean to the left of the median. D, heavy-tailed, has skew -0.19 and excess kurtosis 2.05, with a narrow peak and long tails on both sides.
Figure 2.5. Simulated data (2,000 values per panel). Skew and excess kurtosis were calculated with the same functions that psych::describe() uses. The solid red line is the mean and the dashed line the median: they coincide for symmetric shapes, and the mean is pulled toward the long tail for skewed ones. Panel D has almost no skew but positive excess kurtosis, because both tails are longer than a normal curve would produce.

Figure 2.5 shows the link between shape and summary: when the mean and median separate, the distribution is skewed and the median (IQR) is the better summary. In practice the analyst reads the same evidence from R output rather than from a picture.

5.2.4 Reading summary() and describe() Output

Two functions produce most numerical summaries in this course. Base R's summary() gives the minimum, first quartile, median, mean, third quartile and maximum, plus a count of missing values when there are any. The describe() function from the psych package gives a longer row for each variable, including the standard deviation, skew and excess kurtosis. Worked code 2.11 applies both to three variables that Activity 2.3, at the end of this section, does not use.

Worked code 2.11: annotated summary() and describe() output
summary(phaa$diastolic_bp)
library(psych)
describe(phaa[, c("diastolic_bp", "discrimination_score", "social_support_score")])
Console output
> summary(phaa$diastolic_bp) Min. 1st Qu. Median Mean 3rd Qu. Max. 46.00 64.00 70.00 70.15 76.00 99.00 > library(psych) > describe(phaa[, c("diastolic_bp", "discrimination_score", "social_support_score")]) vars n mean sd median trimmed mad min max diastolic_bp 1 800 70.15 9.12 70 70.12 8.90 46 99 discrimination_score 2 800 7.02 4.03 7 6.99 4.45 0 23 social_support_score 3 780 25.92 6.01 26 25.93 5.93 5 40 range skew kurtosis se diastolic_bp 53 0.07 0.05 0.32 discrimination_score 23 0.13 -0.44 0.14 social_support_score 35 -0.04 -0.29 0.22
ColumnWhat it means
Min., 1st Qu., Median, Mean, 3rd Qu., Max.For diastolic BP the middle half of readings lies between 64 and 76 mmHg (IQR = 12), and the mean (70.15) and median (70) almost coincide, a sign of a symmetric distribution. A seventh column, NA's, appears when values are missing.
nThe number of non-missing values. Social support has 780, so 20 values are missing.
mean, sd, medianThe centre and spread that are reported in a results table.
trimmed, madtrimmed is the mean after dropping the lowest and highest 10% of values, and mad (median absolute deviation) is the median distance of the values from their median, multiplied by 1.4826 so that it equals the SD for normally distributed data. When they are close to the mean and SD, extreme values have little influence.
skew, kurtosisSkewness and excess kurtosis. All three variables have skew between −0.04 and 0.13 and kurtosis between −0.44 and 0.05, so none is noticeably skewed or heavy-tailed.
seThe standard error of the mean, the SD divided by the square root of n. It describes the precision of the mean; the SD is the measure of spread between people.

What to report: because the three variables are close to symmetric, each is summarised as mean (SD): diastolic BP 70.2 (9.1) mmHg, discrimination score 7.0 (4.0) and social support score 25.9 (6.0) with 20 values missing. If |skew| were above about 1, or the histogram showed a long tail, the median and IQR from summary() would be reported instead.

The three variables in Worked code 2.11 are all close to symmetric. The worked example below applies the same reading to output in which one variable is strongly skewed and another is an ordered category, which is the situation in which the choice of summary matters most.

Worked Example 2.7: Choosing Summaries From describe() and table() Output

A hospital survey of 640 patients produces the output below for length of stay (days) and a quality-of-life score (possible range 10 to 50), and a frequency table of education. The numbers are hypothetical.

> describe(dat[, c("los", "qol")]) n mean sd median min max skew kurtosis los 640 6.80 7.90 4.0 1 58 2.41 7.85 qol 612 31.40 6.20 31.0 12 49 -0.08 -0.41 > table(dat$education, useNA = "ifany") Less than high school High school Some college Degree <NA> 58 214 187 141 40

Length of stay. The skew of 2.41 is well above the rule-of-thumb threshold of 1, the excess kurtosis of 7.85 shows much heavier tails than a normal distribution, and the mean (6.8 days) lies far above the median (4.0 days), pulled up by a few very long stays. An SD (7.9) larger than the mean is a further sign of right skew in a variable that cannot fall below zero. Report the median with the IQR (the quartiles come from summary()). A sketch shows a tall peak at short stays, a long tail thinning out to the right towards 58 days, and the median marked to the left of the mean.

Quality of life. The skew (−0.08) is close to 0, the mean (31.4) and median (31.0) almost coincide, and the small negative excess kurtosis (−0.41) means slightly lighter tails and a slightly flatter peak than a normal curve, so the distribution is close to normal. Report the mean and SD, 31.4 (6.2), with the 28 missing values noted. A sketch shows a single, nearly symmetric hump between 12 and 49, centred near 31, with the mean and median at the same point.

Education. The levels are ordered categories, so a mean and SD of the codes have no meaning. Report the count and percentage in each level, in order, among the 600 people with a recorded value (9.7%, 35.7%, 31.2% and 23.5%), and report the 40 missing values. The most common level (the mode) is high school; because the levels are ordered, the median category can also be given, here some college, the level at which the cumulative percentage passes 50%. R gives these numbers directly: round(100 * prop.table(table(dat$education)), 1) prints the percentage in each level among the people with a recorded value, and cumsum() applied to that result prints the cumulative percentages (9.7, 45.3, 76.5 and 100).

The worked example shows that the evidence for a choice of summary sits in the output itself: the skew, the excess kurtosis and the gap between mean and median for a numeric variable, and the variable type for a categorical one. The education row already used a frequency distribution, the main summary for categorical variables, which the next subsection develops.

6.1 Frequency Tables, Cross-Tabulations and the Odds Ratio

For categorical variables, frequency distributions display the count and percentage of observations in each category. Cross-tabulations (contingency tables) display the joint distribution of two categorical variables and are the basis for computing measures of association such as odds ratios and risk ratios. In R, table(x) gives the counts for one variable and table(x, y) cross-tabulates two; prop.table(table(x, y), margin = 1) turns the counts into row proportions, so that each row sums to 1 (100%), and margin = 2 gives column proportions instead. Check which way the percentages run before interpreting them, because the row and column percentages of the same table answer different questions. For example, in a table with smoking status in the rows and hypertension in the columns, row percentages answer "what percentage of smokers have hypertension, compared with non-smokers?", whereas column percentages answer "what percentage of people with hypertension are smokers?" The first compares how common the outcome is in each exposure group, and the second describes who makes up the people with the outcome, so mixing them up leads to a very different conclusion from the same counts.

The most common measure of association from a 2×2 table in epidemiology is the odds ratio. The two worked examples below explain from the start what the number measures. Worked Example 2.8 sets out the cross-tabulation of exposure among cases and controls in an outbreak investigation, shown in Table 2.8, and the odds ratio that R obtains from it.

Worked Example 2.8: Cross-Tabulation in an Outbreak Investigation

Table 2.8. Exposure among 60 cases and 90 controls in an outbreak investigation.

Cases (n)Controls (n)Total
Exposed453075
Unexposed156075
Total6090150

Three out of four cases (45 of 60) were exposed, compared with one in three controls (30 of 90). When these four counts are entered in R, as in Worked code 2.12 below, R prints an odds ratio of 6, suggesting a strong association between exposure and disease.

Worked Example 2.8 gives the number; Worked Example 2.9 explains what it measures. It defines odds and shows how the odds ratio is built from them, with the odds of exposure among cases and among controls set out in Table 2.9.

Worked Example 2.9: From Counts to Odds to an Odds Ratio

The odds of a characteristic compare the number of people who have it with the number who do not: odds = number with / number without. For example, 3 people with for every 1 without gives odds of 3, which corresponds to a probability of 0.75 (3 of every 4 people).

In Table 2.8 the four cells are labelled a = 45 (exposed cases), b = 30 (exposed controls), c = 15 (unexposed cases) and d = 60 (unexposed controls).

Table 2.9. Odds of exposure among the cases and the controls of the outbreak investigation.

GroupExposedUnexposedOdds of exposure (exposed / unexposed)Odds printed by R
Casesa = 45c = 15a / c3
Controlsb = 30d = 60b / d0.5

R prints odds of exposure of 3 among cases and 0.5 among controls (the lines odds_cases and odds_controls in Worked code 2.12): among cases, 3 people were exposed for every 1 who was not, and among controls, 1 person was exposed for every 2 who were not. The odds ratio divides one odds by the other, (a / c) / (b / d), and R prints it as 6. Rearranging gives the cross-product formula, OR = (a × d) / (b × c), which shows that the odds ratio uses all four cells of the table and explains why the same number appears whichever way the ratio is built.

An odds ratio of 1 means the odds are the same in both groups, so there is no association. A value above 1 means the exposure is more common (in odds terms) among cases, and a value below 1 means it is less common. An OR of 6 is a strong association. A descriptive odds ratio makes no allowance for chance or for other differences between the groups; a confidence interval and adjustment for other variables address these.

Together the two worked examples link the number R prints to its meaning: an odds ratio compares the odds of exposure in two groups, and 1 marks no association. In practice the odds and the odds ratio are obtained in R and then interpreted. Worked code 2.12 obtains the outbreak odds ratio in R and applies the same code to hypertension by smoking status in the course data; before reading the result, check which rows and columns of the R table hold the exposed and unexposed cases and controls.

Worked code 2.12: odds and odds ratios from a table

matrix() builds a table from four counts, filling the first column and then the second; dimnames labels the rows and columns. For the course data, table() builds the same kind of table from two variables, and tab["Yes", "No"] picks the cell in the row labelled "Yes" and the column labelled "No".

# The outbreak table from the text
outbreak <- matrix(c(45, 15, 30, 60), nrow = 2,
                   dimnames = list(Exposure = c("Exposed", "Unexposed"),
                                   Group    = c("Case", "Control")))
outbreak
odds_cases    <- outbreak["Exposed", "Case"] / outbreak["Unexposed", "Case"]
odds_controls <- outbreak["Exposed", "Control"] / outbreak["Unexposed", "Control"]
odds_cases
odds_controls
odds_cases / odds_controls              # the odds ratio

# The same steps with the course data: hypertension by smoking status
tab <- table(Smoker = phaa$smoker, Hypertension = phaa$hypertension)
tab
odds_smokers     <- tab["Yes", "Yes"] / tab["Yes", "No"]
odds_non_smokers <- tab["No", "Yes"] / tab["No", "No"]
round(c(smokers = odds_smokers, non_smokers = odds_non_smokers,
        OR = odds_smokers / odds_non_smokers), 3)
Console output
> outbreak <- matrix(c(45, 15, 30, 60), nrow = 2, + dimnames = list(Exposure = c("Exposed", "Unexposed"), + Group = c("Case", "Control"))) > outbreak Group Exposure Case Control Exposed 45 30 Unexposed 15 60 > odds_cases <- outbreak["Exposed", "Case"] / outbreak["Unexposed", "Case"] > odds_controls <- outbreak["Exposed", "Control"] / outbreak["Unexposed", "Control"] > odds_cases [1] 3 > odds_controls [1] 0.5 > odds_cases / odds_controls [1] 6 > tab <- table(Smoker = phaa$smoker, Hypertension = phaa$hypertension) > tab Hypertension Smoker No Yes No 649 15 Yes 128 8 > odds_smokers <- tab["Yes", "Yes"] / tab["Yes", "No"] > odds_non_smokers <- tab["No", "Yes"] / tab["No", "No"] > round(c(smokers = odds_smokers, non_smokers = odds_non_smokers, + OR = odds_smokers / odds_non_smokers), 3) smokers non_smokers OR 0.062 0.023 2.704

The outbreak calculation reproduces the odds of 3 and 0.5 and the odds ratio of 6. In the course data, 8 of 136 smokers and 15 of 664 non-smokers have hypertension, giving odds of 0.062 and 0.023 and an odds ratio of 2.70: the odds of hypertension are about 2.7 times as high among smokers. With only 23 people with hypertension this estimate is imprecise; a logistic regression would add a confidence interval and adjust for age and other variables.

The course-data example shows that the same two lines of R give the odds ratio for any 2×2 table, provided the cells are read in the right order. A cross-tabulation compares two groups on one characteristic; the next subsection brings many such comparisons together in a single table.

6.2 Constructing a Table 1 Stratified by Exposure

6.2.1 Setting Out a Table 1

In epidemiologic publications, “Table 1” typically presents the baseline characteristics of the study population, often stratified by exposure or outcome status. How baseline characteristics and group comparisons are reported has been the subject of methodological scrutiny. Pocock, Assmann, Enos, & Kasten (2002) reviewed published trial reports and catalogued common problems with baseline comparisons, subgroup analyses and covariate adjustment. A well-constructed Table 1 includes:

  • Demographics (age, sex, ethnicity) with appropriate summary statistics
  • Key clinical or exposure variables
  • Missing data counts for each variable
  • A comparison column between groups, preferably standardised differences (explained below)

Setting out the structure of a Table 1 takes three decisions. The columns are the groups being compared, usually the exposure groups (for example, any physical activity versus none), each headed by its number of participants, often with an overall column; the variable that defines the columns is therefore not also a row. The rows are the characteristics that matter for the analysis: demographics, the potential confounders and other covariates, and, if wanted, the outcome. The statistic in each cell follows the reporting convention: mean (SD) for approximately normal continuous variables, median (IQR) for skewed ones, and n (%) for each level of a categorical variable, with the number missing reported for every variable that has gaps.

Table 2.10 shows the structure of a Table 1 for a hypothetical study that compares exposed and unexposed participants. As you read it, check each row's statistic against the reporting convention.

Table 2.10. The structure of a Table 1 for a hypothetical study comparing exposed and unexposed participants, in a layout common in published papers.

CharacteristicExposed (n=200)Unexposed (n=300)p-value
Age, mean ± SD52.3 ± 11.449.8 ± 12.10.02
Female, n (%)110 (55.0%)162 (54.0%)0.82
BMI, median (IQR)27.1 (24.0–31.5)25.8 (23.2–29.4)0.004
Current smoker, n (%)48 (24.0%)51 (17.0%)0.05
Diabetes, n (%)34 (17.0%)30 (10.0%)0.02

In Table 2.10 the continuous variable with a normal distribution (age) uses mean ± SD, while the skewed variable (BMI) uses median (IQR), and the categorical variables are presented as n (%). In this hypothetical study BMI was right-skewed, so its median (IQR) was reported. In the course data BMI is close to symmetric (skew 0.17 in Activity 2.3), so the course-data table in Worked code 2.13 reports it as mean (SD). The choice follows the distribution of each variable in the data at hand. Table 2.10 also follows a layout common in published papers, with a p-value column and without an overall column or counts of missing values; the course-data table in Worked code 2.13 shows the layout this lesson recommends.

6.2.2 The Comparison Column: p-Values and Standardised Differences

The p-value column of Table 2.10 raises the question of how the groups should be compared. The definitions and the recommendation below are worth committing to memory, because they decide how every Table 1 in this course is read.

Null Hypotheses, p-Values and Standardised Differences

A null hypothesis is the starting assumption of no difference: here, that exposed and unexposed people have the same average age in the population the sample came from. A p-value is the probability of seeing a difference at least as large as the one observed if the null hypothesis were true. A small p-value (conventionally below 0.05) means the observed difference would be surprising if there were truly no difference between the groups.

A standardised difference (also called the standardised mean difference, SMD) expresses the gap between two groups in standard-deviation units. For a continuous variable it is the difference in means divided by the square root of the average of the two groups' variances; for a proportion, the difference in proportions is divided by a standard deviation calculated from the two proportions. An SMD of 0.5 means the groups differ by half a standard deviation. Dividing by a standard deviation puts every variable on the same scale. By this we mean that a difference in age measured in years, a difference in blood pressure measured in mmHg and a difference in the percentage of women can all be compared directly, because each is expressed relative to how much that variable usually varies between people. Values below about 0.1 are commonly read as a negligible imbalance, although this cut-off is a convention.

Recommendation for this course. Table 1 describes the sample, so this course follows the STROBE reporting guideline for observational studies, which advises that significance tests be avoided in descriptive tables (Vandenbroucke et al., 2007). A p-value in Table 1 depends heavily on sample size and describes chance variation; whether an imbalance is large enough to matter as a confounder (a variable that can distort the association between exposure and outcome, defined more fully in 6.2.4) is a separate question, which the standardised difference addresses more directly. Where a comparison column is wanted, report the standardised difference. When a journal requires p-values, they should be read as descriptions of chance variation.

In short, a p-value answers a question about chance that depends on the sample size, whereas a standardised difference measures the size of the imbalance itself. The difference shows up as soon as the sample size changes. For example, the same gap of two years in average age might give a large p-value in a study of 100 people and a very small p-value in a study of 10,000 people, because a larger sample makes it easier to rule out chance. The standardised difference for that gap would be about the same in both studies, because it depends only on the size of the gap relative to the spread of ages, not on how many people were measured. This is why the standardised difference is the better guide to whether an imbalance is large enough to matter.

6.2.3 A Table 1 for the Course Data

Worked code 2.13 puts the three decisions and the standardised difference into practice. It builds a Table 1 for the course data, stratified by smoking status, with an overall column, standardised differences and counts of missing values, and then turns the R output into the report table shown as Table 2.11.

Worked code 2.13: A Table 1 for the course data, stratified by smoking status

Base R can build Table 1 with three small helper functions. function(v) { ... } defines a reusable recipe, tapply(v, grp, f) applies the recipe f separately to each smoking group, and sprintf() formats numbers as text: %.1f means one decimal place, %d a whole number and %% a percent sign. The comparison phaa$education >= "Bachelor's" works because education was made an ordered factor in the cleaning activity; it is TRUE for a bachelor's or graduate degree.

# Three small helper functions. function(v) { ... } defines a reusable recipe
# that is applied to the whole sample and then to each smoking group.
grp <- phaa$smoker                       # the grouping variable ("No", "Yes")

mean_sd <- function(v) {                 # "mean (SD)" and the standardised difference
  f <- function(x) sprintf("%.1f (%.1f)", mean(x, na.rm = TRUE), sd(x, na.rm = TRUE))
  m <- tapply(v, grp, mean, na.rm = TRUE)
  s <- tapply(v, grp, sd, na.rm = TRUE)
  d <- (m["Yes"] - m["No"]) / sqrt((s["Yes"]^2 + s["No"]^2) / 2)
  c(Overall = f(v), tapply(v, grp, f), SMD = sprintf("%.2f", d))
}
n_pct <- function(v, level) {            # "n (%)" and the standardised difference
  f <- function(x) sprintf("%d (%.1f%%)", sum(x == level, na.rm = TRUE),
                           100 * mean(x == level, na.rm = TRUE))
  p <- tapply(v == level, grp, mean, na.rm = TRUE)
  d <- (p["Yes"] - p["No"]) / sqrt((p["Yes"] * (1 - p["Yes"]) + p["No"] * (1 - p["No"])) / 2)
  c(Overall = f(v), tapply(v, grp, f), SMD = sprintf("%.2f", d))
}
n_miss <- function(v) {                  # number of missing values
  c(Overall = sum(is.na(v)), tapply(is.na(v), grp, sum), SMD = "")
}

table1 <- rbind(
  "N"                     = c(Overall = length(grp), table(grp), SMD = ""),
  "Age, mean (SD)"        = mean_sd(phaa$age),
  "  Age missing, n"      = n_miss(phaa$age),
  "Woman, n (%)"          = n_pct(phaa$gender, "Woman"),
  "Degree, n (%)"         = n_pct(phaa$education >= "Bachelor's", TRUE),
  "BMI, mean (SD)"        = mean_sd(phaa$bmi),
  "SBP, mean (SD)"        = mean_sd(phaa$systolic_bp),
  "Activity, mean (SD)"   = mean_sd(phaa$phys_act_min),
  "  Activity missing, n" = n_miss(phaa$phys_act_min),
  "Income missing, n (%)" = n_pct(is.na(phaa$income), TRUE)
)
noquote(table1)
Console output
> noquote(table1) Overall No Yes SMD N 800 664 136 Age, mean (SD) 45.1 (13.6) 45.1 (13.4) 45.3 (14.9) 0.02 Age missing, n 2 1 1 Woman, n (%) 414 (51.7%) 340 (51.2%) 74 (54.4%) 0.06 Degree, n (%) 360 (45.0%) 304 (45.8%) 56 (41.2%) -0.09 BMI, mean (SD) 20.8 (3.2) 21.0 (3.2) 19.5 (2.9) -0.50 SBP, mean (SD) 108.2 (11.3) 107.2 (11.0) 113.5 (11.6) 0.56 Activity, mean (SD) 176.1 (82.4) 176.0 (83.4) 176.1 (77.1) 0.00 Activity missing, n 28 20 8 Income missing, n (%) 35 (4.4%) 32 (4.8%) 3 (2.2%) -0.14

The output has one column for the whole sample, one for non-smokers ("No"), one for smokers ("Yes") and the standardised difference (smokers minus non-smokers). Smokers and non-smokers have nearly the same age (SMD 0.02), proportion of women (0.06), education (−0.09) and physical activity (0.00). Smokers have a lower mean BMI (19.5 versus 21.0 kg/m², SMD −0.50) and a higher mean systolic BP (113.5 versus 107.2 mmHg, SMD 0.56). BMI differs between the groups, but smoking can itself lower body weight, so BMI may be a mediator, a variable on the causal path from smoking to blood pressure. In the simulated course data BMI was generated to depend on smoking status. Adjusting for a mediator removes part of the effect being estimated, so whether BMI enters a model is decided from subject knowledge about causal order, which the regression lessons will need to consider. The missing-value rows show that missing activity data are spread across both groups (20 and 8).

Turning the R output into a report table involves relabelling the columns (with group sizes in the headers), writing units and full names into the row labels, removing the percent signs that the row label already states, and adding a footnote that defines abbreviations. The numbers themselves are copied unchanged, as Table 2.11 shows.

Table 2.11. Characteristics of the course sample, overall and by smoking status, set out for a report.

CharacteristicOverall (n = 800)Non-smokers (n = 664)Smokers (n = 136)Std. diff.
Age (years), mean (SD)45.1 (13.6)45.1 (13.4)45.3 (14.9)0.02
  Missing, n211
Women, n (%)414 (51.7)340 (51.2)74 (54.4)0.06
Bachelor's degree or higher, n (%)360 (45.0)304 (45.8)56 (41.2)−0.09
BMI (kg/m²), mean (SD)20.8 (3.2)21.0 (3.2)19.5 (2.9)−0.50
Systolic BP (mmHg), mean (SD)108.2 (11.3)107.2 (11.0)113.5 (11.6)0.56
Physical activity (min/week), mean (SD)176.1 (82.4)176.0 (83.4)176.1 (77.1)0.00
  Missing, n28208
Income not reported, n (%)35 (4.4)32 (4.8)3 (2.2)−0.14

Table note for a report: "Std. diff.: standardised difference, smokers minus non-smokers. SD: standard deviation. BMI: body mass index. One BMI value and two systolic BP values were set to missing by range checks."

Table 2.11 is a complete Table 1: each column is headed by its n, each row carries the statistic that suits it, the missing values are reported, and the comparison column holds standardised differences. A Table 1 stratified by exposure shows how characteristics are distributed across the exposure groups, and the next part explains why that distribution matters before any multivariable model is fitted.

6.2.4 Stratification and Confounding

Stratifying descriptive statistics by key exposure or covariate groups allows you to assess the distribution of potential confounders across exposure categories. By this we mean that each characteristic is summarised separately within each exposure group, so that the groups can be compared side by side, as the columns of Table 2.11 do for smokers and non-smokers. This is a critical step before proceeding to multivariable modelling, as it helps identify imbalances that may require adjustment. If the exposure groups differ on a characteristic that also affects the outcome, a simple comparison of the outcome between the groups mixes the effect of the exposure with the effect of that characteristic.

A confounder is a third variable that is related to both the exposure and the outcome and can therefore make the two appear more, or less, strongly associated than they truly are. Age is a common example, because it is linked to many exposures and to blood pressure. For example, if smokers in a sample were on average much older than non-smokers, smokers would tend to have higher blood pressure partly because of their age, and a crude comparison would overstate the association between smoking and blood pressure. A large standardised difference in Table 1 marks a variable that differs between the exposure groups; whether it also affects the outcome is judged from subject knowledge and the later regression models. A variable that is itself affected by the exposure is a mediator and does not meet the definition of a confounder. The difference lies in causal order. A confounder exists before the exposure and influences both the exposure and the outcome, whereas a mediator comes after the exposure and carries part of its effect to the outcome. Adjusting for a confounder removes a distortion, but adjusting for a mediator removes part of the very effect being estimated.

In the course data, smoking status is the exposure, and the large standardised differences for BMI and systolic BP in Table 2.11 are the variables to think about before any model is fitted. Tables summarise the groups in numbers; the next subsection turns to the plots that show the same data as pictures.

7.1 Choosing a Visualisation

Appropriate visualization depends on the variable type and the question being asked. Exploratory data analysis emphasises looking at data first so structure, gaps, and anomalies are visible before any model is fit. Four kinds of plot cover most needs: a histogram for the shape of one continuous variable, side-by-side boxplots for a continuous variable compared across groups, a bar chart for the frequencies of a categorical variable, and a scatter plot for two continuous variables (or a line graph for a trend over time). Open each tab below to see what each plot shows, when to use it and when not to, with an example from the course data or a simulation.

Histograms display the distribution of a continuous variable by dividing values into bins and counting observations per bin. They reveal distributional shape, modality, gaps, and potential outliers. The choice of bin width affects interpretation: too few bins obscure patterns, too many create noise.

Figure 2.6 shows the same 850 simulated readings, drawn from two groups with different average blood pressures, using 5, 20 and 100 bins. In R the number of bins is suggested with the breaks argument, for example hist(phaa$systolic_bp, breaks = 20); R treats the number as a suggestion and chooses round cut-points near it.

Three histograms of the same simulated systolic blood pressure data. With 5 bins the distribution looks like a single lumpy shape. With 20 bins two peaks appear, near 115 and 150 mmHg. With 100 bins the bars are jagged and the two peaks are hard to separate from random variation.
Figure 2.6. Simulated data. A: with 5 bins the two groups merge into one shape. B: with 20 bins the two peaks near 115 and 150 mmHg are clear. C: with 100 bins each bar holds only a few people and the outline is mostly noise. Trying two or three bin numbers before interpreting a histogram is good practice.

Boxplots display the median, the IQR (the box), whiskers that extend to the most extreme observation lying within 1.5×IQR of the box, and individual points beyond the whiskers. They are ideal for comparing distributions across groups (e.g., blood pressure by treatment arm) and for quickly identifying asymmetry and outliers.

Figure 2.7 shows an annotated boxplot of systolic blood pressure in the course data after the range checks.

Annotated boxplot of systolic blood pressure in the course data. The box runs from Q1 = 100 to Q3 = 116 mmHg with the median at 109. The upper whisker ends at 140 and the lower whisker at 85. Two points, 142 and 145 mmHg, lie above the upper whisker.
Figure 2.7. Course data after the range checks (n = 798). The whiskers stop at the most extreme readings inside the fences (Q1 − 1.5 × IQR = 76 and Q3 + 1.5 × IQR = 140), which here are 85 and 140 mmHg. Readings beyond the fences are drawn as individual points.

Bar charts display frequencies or proportions for categorical variables. Use them for comparing counts across groups. Avoid using bar charts for continuous data (use histograms instead) and avoid 3D bar charts, which distort proportions.

Scatter plots display the relationship between two continuous variables and can reveal linear or nonlinear associations, clusters, and outliers. Line graphs are appropriate for displaying trends over time (e.g., incidence rates by year). Both are essential in epidemiologic exploratory analysis.

Figure 2.8 plots systolic blood pressure against age in the course data.

Scatter plot of systolic blood pressure against age in the course data. The cloud of points rises from lower left to upper right.
Figure 2.8. Course data. Each point is one participant. The upward drift of the cloud shows that older participants tend to have higher systolic BP. The correlation, a measure of linear association that runs from −1 to 1, is 0.54. No point sits far from the cloud, so there is no sign of a bivariate outlier.

The tabs share one rule: pick the plot by asking what kind of variable you have and what question the picture should answer. A 3D bar chart, or a bar chart of group means, hides the spread and outliers that a group comparison should show. Activity 2.3, at the end of this section, draws the four standard plots described in the tabs for the course data: a histogram, side-by-side boxplots by smoking status, a bar chart and a scatter plot. Always plot before you fit a regression. A boxplot of systolic_bp by smoker tells you in a second whether a regression comparing the two groups will be driven by a few extreme values or by a real shift in the centre of the distribution.

A histogram also gives a first answer to a question that decides which methods are appropriate later: whether a variable is close enough to normal. The next subsection makes that judgement systematic.

7.2 Assessing Normality

7.2.1 Visual Checks and Formal Tests

Assessing whether a variable follows a normal distribution informs the choice between parametric and non-parametric methods. There are two kinds of evidence: visual methods, chiefly the normal Q-Q plot, and formal tests such as the Shapiro-Wilk test. Expand each panel below to see how to read a Q-Q plot and what a test's p-value can and cannot tell you; the caveat about very large samples in the second panel is the point most often missed.

Visual Methods

Q-Q (quantile-quantile) plots compare observed quantiles to theoretical normal quantiles. Points falling along the diagonal line suggest normality; systematic departures indicate non-normality. Histograms with a normal curve overlay also provide quick visual assessment.

In plain terms, a Q-Q plot sorts the data from smallest to largest and plots each value against the z-score that a standard normal distribution would place at the same position. These z-scores form the horizontal axis, labelled theoretical quantiles, which runs from about −3 to 3. If the data are normal, the points fall along a straight line. Points that curve away from the line at one end indicate skew, and points that bend away at both ends in an S shape indicate heavy tails, meaning more extreme values than a normal distribution would produce. In R, qqnorm(x) draws the points and qqline(x) adds the reference line.

Formal Tests

The Shapiro-Wilk test is generally preferred for samples under 5,000. The Kolmogorov-Smirnov test (with Lilliefors correction) is an alternative. Both test the null hypothesis that data come from a normal distribution.

Because the null hypothesis is normality, a small p-value (below 0.05 by convention) is evidence of a departure from normality, and a large p-value means that no departure was detected. The p-value also depends on sample size: the same small departure gives a large p-value in a sample of 50 and a very small one in a sample of 50,000, because a test with many observations has the power to detect even trivial departures. R's shapiro.test() accepts between 3 and 5,000 values.

Caveat: With very large samples, these tests will reject normality even for trivial departures. Visual assessment should always accompany formal tests.

The three patterns described in the first panel are easiest to learn by sight. Figure 2.9 shows histograms with their normal Q-Q plots for a normal, a right-skewed and a heavy-tailed distribution; compare where each set of points leaves the line.

Three histograms with their normal Q-Q plots beneath. A, normal: the Q-Q points follow the red line. B, right-skewed: the points curve above the line at the right end and sit above it at the left end. C, heavy-tailed: the points lie on the line in the middle and bend away at both ends.
Figure 2.9. Simulated data (300 values per panel). A: a normal distribution, whose points follow the line. B: a right-skewed distribution, whose points curve away at the right end. C: a heavy-tailed distribution, whose points bend away at both ends (low values below the line, high values above it).

Worked code 2.14 applies both kinds of evidence to the course data: it draws a Q-Q plot of systolic blood pressure and applies the Shapiro-Wilk test to systolic and diastolic blood pressure. Watch how the test result for systolic BP compares with its plot and skew.

Worked code 2.14: a Q-Q plot and the Shapiro-Wilk test
qqnorm(phaa$systolic_bp, main = "Systolic BP: normal Q-Q plot")
qqline(phaa$systolic_bp)          # the reference line
shapiro.test(phaa$systolic_bp)
shapiro.test(phaa$diastolic_bp)
Console output
> qqnorm(phaa$systolic_bp, main = "Systolic BP: normal Q-Q plot") > qqline(phaa$systolic_bp) > shapiro.test(phaa$systolic_bp) Shapiro-Wilk normality test data: phaa$systolic_bp W = 0.99258, p-value = 0.0005134 > shapiro.test(phaa$diastolic_bp) Shapiro-Wilk normality test data: phaa$diastolic_bp W = 0.99672, p-value = 0.09998
OutputWhat it shows
WThe test statistic, which is close to 1 when the data look normal. Both values (0.993 and 0.997) are very close to 1.
Systolic BP: p-value = 0.0005The small p-value means a departure from normality was detected. In the Q-Q plot the points follow the line closely except at the lower end, where the 18 lowest readings are all exactly 85 mmHg and form a short flat run. With 798 readings the test detects this small departure, while the skew of 0.05 shows that the distribution is close to symmetric.
Diastolic BP: p-value = 0.10No departure from normality was detected.

Systolic BP shows the large-sample problem at first hand: the test detects a departure (p = 0.0005), yet the plot is close to a straight line and the skew is only 0.05, so the departure is too small to matter.

7.2.2 From the Checks to a Decision

In practice the analyst combines the plot, the skew value and the purpose of the analysis. If the Q-Q plot is close to a straight line and |skew| is below about 1, the variable is summarised as mean (SD) and parametric methods are reasonable, even when a large sample makes the Shapiro-Wilk p-value small, as for systolic BP here. If the plot curves clearly or |skew| is above about 1, the variable is summarised as median (IQR), and the analyst either transforms it (as in the previous section) or uses a rank-based method. Rank-based methods replace each value by its rank (1 for the smallest, 2 for the next, and so on), so a few extreme values cannot dominate the result. Table 2.12 pairs the common options; for each question, read across from the parametric method to its rank-based alternative.

Table 2.12. Parametric and rank-based options for three common analytic questions, with the R function for each.

QuestionParametric option (assumes approximate normality)Rank-based option (fewer assumptions)
Do two groups differ?The t-test compares the two means: t.test(y ~ group). R uses the Welch version by default, which allows the two groups to have different spreads.The Wilcoxon rank-sum test compares the ranks of the two groups: wilcox.test(y ~ group).
Do three or more groups differ?One-way analysis of variance (ANOVA) compares the means: summary(aov(y ~ group)).The Kruskal-Wallis test compares the ranks: kruskal.test(y ~ group).
Are two continuous variables associated?Pearson correlation measures linear association: cor.test(x, y).Spearman correlation is Pearson correlation calculated on the ranks: cor.test(x, y, method = "spearman").

Table 2.12 sums up the decision: if a variable is approximately normal, the parametric methods in the middle column are appropriate, and for clearly non-normal distributions the rank-based alternatives in the right-hand column, or a transformation, are the safer choice. The Central Limit Theorem provides some protection for means in large samples, but descriptive summaries should still reflect the actual distribution. The Central Limit Theorem states that the means of repeated large samples follow an approximately normal distribution even when the individual values do not. By this we mean that a t-test or confidence interval for a mean can still work reasonably well with a skewed variable once the sample is large, because those methods rely on the behaviour of the mean rather than of single values. This protection applies to inference about the mean only. It does not make a skewed variable symmetric, so a Table 1 or a descriptive summary of that variable should still use the median (IQR) when the distribution is clearly skewed.

Additional Context

Building Scales: Internal Consistency and Factor Analysis

This part is background for Activity 2.3 and for analyses that use the depression score, rather than an exam objective. Many public-health surveys ask several items that together measure a latent construct (e.g., depression, anxiety, social support). Two questions need to be answered before you can use these items as a single scale variable in a regression: (1) internal consistency (do the items hang together?), assessed with Cronbach’s α; and (2) dimensionality (how many underlying factors do the items measure?), assessed with exploratory factor analysis (EFA). Once both checks pass, the items can be summed (or averaged) into a derived scale variable that you carry forward into later analyses.

Each depression item asks how often a symptom occurred, on a five-point response scale from 1 (Never) to 5 (Always), so a higher number always means more symptoms. Some questionnaires also include reverse-coded items, worded in the opposite direction (for example, "I felt hopeful about the future"), where a high answer means fewer symptoms. Such items must be flipped before alpha is calculated or a score is built; on a 1 to 5 scale the flipped value is 6 minus the original, so 5 becomes 1 and 1 becomes 5. The alpha() function prints a warning when an item correlates negatively with the rest of the scale, which is the usual sign of an item that has not been reversed. None of the course items is reverse-coded.

Cronbach's alpha can be understood as a measure of how well the items agree. If people who score high on one item tend to score high on the others, the items share a common source, the construct, and alpha is high. Alpha rises when the items are more strongly correlated with each other and when there are more of them. Under the assumptions behind it, an alpha of 0.90 indicates that about 90% of the variation in total scores reflects what the items have in common and about 10% reflects item-specific noise. Values of about 0.70 or higher are conventionally acceptable and 0.80 or higher good; values above about 0.95 can mean that some items are near-duplicates.

Two terms from factor analysis appear in the output. An eigenvalue measures how much of the shared variation among the items one factor accounts for; a large first eigenvalue followed by small ones suggests a single underlying factor. Parallel analysis compares each eigenvalue from the real data with the eigenvalue that random data of the same size would produce, and keeps factors, counting from the first, for as long as the real eigenvalue is larger. A factor loading is the correlation between an item and a factor; loadings of 0.40 or higher are conventionally treated as meaningful, and this lesson uses that threshold throughout.

Activity 2.3, at the end of this section, applies both checks to the seven depression items (dep1 to dep7) and five anxiety items (anx1 to anx5) in the course data: it confirms internal consistency with alpha(), decides on the number of factors with fa.parallel() and factanal(), and builds the derived scale scores. The reference code below prints the full output of these functions for the depression items; the table after it explains each part, and Figure 2.10 annotates the parallel-analysis plot.

Reference code: the full output of alpha(), fa.parallel() and factanal() for the depression items
# 1. Internal consistency for each candidate scale ----------------------------
library(psych)
dep_items <- phaa[, c("dep1","dep2","dep3","dep4","dep5","dep6","dep7")]
anx_items <- phaa[, c("anx1","anx2","anx3","anx4","anx5")]

alpha(dep_items)             # raw_alpha >= 0.70 is acceptable, >= 0.80 is good
alpha(anx_items)
dep_alpha <- alpha(dep_items)            # store the result to read it at 3 decimals
round(dep_alpha$total$raw_alpha, 3)
round(dep_alpha$alpha.drop[, "raw_alpha", drop = FALSE], 3)  # alpha if an item is dropped

# 2. How many underlying factors? ------------------------------------------
fa.parallel(dep_items, fa = "fa")   # parallel analysis for factors

# 3. Inspect a one-factor solution for the depression items ----------------
factanal(x = na.omit(dep_items), factors = 1)
# na.omit(): factanal() needs complete rows and dep4 has 15 NAs, so these 15
# people are left out of this check (a complete-case analysis).
# With a single factor there is nothing to rotate, so no rotation is requested.
# Loadings >= 0.40 contribute meaningfully to the factor.

# 4. Two-factor solution combining dep + anx items -------------------------
combined <- na.omit(cbind(dep_items, anx_items))
factanal(x = combined, factors = 2, rotation = "varimax")
# dep1-7 should load on one factor; anx1-5 on the other.

# 5. Build derived scale variables for later analyses ------------------------
# Score = mean of the answered items x number of items, calculated only when
# at most one item is missing (otherwise NA). A missing item is never
# counted as 0, which would understate the person's score.
n_dep <- rowSums(!is.na(dep_items))     # items answered by each person
n_anx <- rowSums(!is.na(anx_items))
phaa$dep_score <- ifelse(n_dep >= 6, rowMeans(dep_items, na.rm = TRUE) * 7, NA)
phaa$anx_score <- ifelse(n_anx >= 4, rowMeans(anx_items, na.rm = TRUE) * 5, NA)
summary(phaa$dep_score)
Console output (depression items; Activity 2.3 adds the anxiety items and the two-factor solution)
> alpha(dep_items) Reliability analysis Call: alpha(x = dep_items) raw_alpha std.alpha G6(smc) average_r S/N ase mean sd median_r 0.9 0.9 0.89 0.57 9.4 0.0052 3 1.1 0.58 95% confidence boundaries lower alpha upper Feldt 0.89 0.9 0.91 Duhachek 0.89 0.9 0.91 Reliability if an item is dropped: raw_alpha std.alpha G6(smc) average_r S/N alpha se var.r med.r dep1 0.89 0.89 0.87 0.57 8.0 0.0061 0.00111 0.57 dep2 0.89 0.89 0.87 0.56 7.8 0.0062 0.00118 0.57 dep3 0.89 0.89 0.87 0.57 7.9 0.0062 0.00157 0.57 dep4 0.89 0.89 0.87 0.57 7.8 0.0062 0.00132 0.57 dep5 0.89 0.89 0.87 0.58 8.3 0.0059 0.00126 0.59 dep6 0.89 0.89 0.87 0.57 8.0 0.0061 0.00129 0.57 dep7 0.90 0.90 0.88 0.59 8.6 0.0057 0.00042 0.59 Item statistics n raw.r std.r r.cor r.drop mean sd dep1 800 0.80 0.80 0.76 0.72 3 1.4 dep2 800 0.82 0.82 0.78 0.74 3 1.4 dep3 800 0.81 0.81 0.77 0.73 3 1.4 dep4 785 0.81 0.81 0.78 0.74 3 1.4 dep5 800 0.78 0.78 0.73 0.69 3 1.4 dep6 800 0.80 0.80 0.75 0.72 3 1.4 dep7 800 0.75 0.75 0.69 0.65 3 1.4 ... (10 more lines not shown) > round(dep_alpha$total$raw_alpha, 3) [1] 0.904 > round(dep_alpha$alpha.drop[, "raw_alpha", drop = FALSE], 3) raw_alpha dep1 0.889 dep2 0.886 dep3 0.888 dep4 0.887 dep5 0.892 dep6 0.889 dep7 0.896 > fa.parallel(dep_items, fa = "fa") Parallel analysis suggests that the number of factors = 1 and the number of components = NA > factanal(x = na.omit(dep_items), factors = 1) Call: factanal(x = na.omit(dep_items), factors = 1) Uniquenesses: dep1 dep2 dep3 dep4 dep5 dep6 dep7 0.405 0.368 0.401 0.391 0.465 0.417 0.523 Loadings: Factor1 dep1 0.772 dep2 0.795 dep3 0.774 dep4 0.781 dep5 0.731 dep6 0.763 dep7 0.691 Factor1 SS loadings 4.030 Proportion Var 0.576 Test of the hypothesis that 1 factor is sufficient. The chi square statistic is 12.7 on 14 degrees of freedom. The p-value is 0.551
OutputWhat it shows and what to report
raw_alphaCronbach's alpha, printed to two decimals as 0.9. The line round(dep_alpha$total$raw_alpha, 3) gives 0.904, which is the value to report (α = 0.904).
std.alpha, average_rAlpha calculated from standardised items (similar here because all items share the same response scale), and the average correlation between pairs of items (0.57).
95% confidence boundariesThe range of alpha values compatible with the data, 0.89 to 0.91.
Reliability if an item is droppedEach row gives alpha with that item removed. A value above the overall 0.904 would mean the scale is better without the item. Every value here is lower (0.886 to 0.896 at three decimals), so every item contributes; dep7 contributes least, because removing it would lower alpha only to 0.896.
r.drop (item statistics)The correlation of each item with the total of the other items, from 0.65 (dep7) to 0.74. Values below about 0.30 would mark an item that does not fit the scale.
fa.parallel()The message reports one factor (the number of components is NA because fa = "fa" requests factors only). The annotated plot below (Figure 2.10) shows why.
factanal() LoadingsAll seven loadings lie between 0.691 (dep7) and 0.795 (dep2), well above 0.40. Uniquenesses are the share of each item that the factor does not explain (1 minus the squared loading).
Proportion Var and the chi-square testThe single factor explains 57.6% of the item variance. The test asks whether one factor is enough; its large p-value (0.551) gives no evidence that more factors are needed.
Parallel analysis plot for the seven depression items. The blue line with triangles shows the eigenvalues from the real data: 4.02 for factor 1, then values near zero for factors 2 to 7. Two red lines show eigenvalues from random data of the same size, starting near 0.5 and falling toward zero. Factor 1 lies far above the red lines and factor 2 falls below them, so one factor is kept.
Figure 2.10. Course data. The blue line (FA Actual Data) shows the eigenvalues of the real items; the red dotted and dashed lines (FA Simulated Data and FA Resampled Data) show the eigenvalues that random data of the same size produce. Counting from factor 1, factors are kept until the first blue point that falls below the red lines. That happens at factor 2, so one factor is kept, and the near-zero eigenvalues of later factors play no part. The red lines vary slightly from run to run because they come from random data.

How to read this. An α of 0.904 indicates that the seven depression items measure essentially the same construct. The single factor explains 57.6% of the item variance and every loading is well above the 0.40 threshold (the lowest is 0.691, for dep7), so combining the items into dep_score is justified. Step 5 builds the score as the mean of the answered items multiplied by the number of items (a prorated sum), and only for people who answered at least six of the seven depression items (four of the five anxiety items). Using rowSums(dep_items, na.rm = TRUE) instead would count a skipped item as 0 and understate the scores of the 15 people who skipped dep4, by between 1.8 and 4.5 points (2.8 on average). The score can then serve as a continuous predictor, or as a covariate in a logistic model for hypertension.

Both checks pass for the depression items, so the derived dep_score can be carried forward as a single variable. Activity 2.3 practises everything in this section on the cleaned course data: numeric and categorical summaries, a cross-tabulation and an odds ratio, a Table 1 stratified by smoking status, the four standard plots, a normality check, and the reliability and factor structure of the depression and anxiety items. Run it along with the box and use your own output to answer the questions at the end.

R Activity 2.3: describe the course data and check the scales

Text version of the activity: the code of the run-along script, its console output and notes on reading it.

phaa <- read.csv("https://heal-learn.ca/r-activities/data/phaa_survey_clean.csv")   # course data, read from the website
phaa$smoker <- factor(phaa$smoker, levels = c("No", "Yes"))
phaa$income <- factor(phaa$income, levels = c("<25k", "25-50k", "50-75k", "75-100k", ">100k"))
library(psych)      # describe(), alpha() and fa.parallel()
library(tableone)   # CreateTableOne() for Table 1

The cleaned file is the survey after the range checks of Section 2, with impossible ages, blood pressures and BMI values set to NA. The income bands are put in order from lowest to highest so that tables list them in that order. If R says that there is no package called psych or tableone, run install.packages(c("psych", "tableone")) once.

summary(phaa$age)
describe(phaa[, c("age", "bmi", "systolic_bp", "phys_act_min")])
table(phaa$income, useNA = "ifany")     # ordered categories
Console output
Min. 1st Qu. Median Mean 3rd Qu. Max. NAs 18.00 36.00 45.00 45.15 54.00 87.00 2 vars n mean sd median trimmed mad min max range skew age 1 798 45.15 13.62 45.0 45.01 13.34 18 87.0 69.0 0.12 bmi 2 799 20.77 3.24 20.7 20.73 3.56 15 31.6 16.6 0.17 systolic_bp 3 798 108.22 11.34 109.0 108.23 11.86 85 145.0 60.0 0.05 phys_act_min 4 772 176.05 82.37 176.5 175.31 81.54 0 479.0 479.0 0.16 kurtosis se age -0.36 0.48 bmi -0.41 0.11 systolic_bp -0.31 0.40 phys_act_min -0.01 2.96 <25k 25-50k 50-75k 75-100k >100k <NA> 94 159 207 170 135 35

Reading the output. For each numeric variable compare the mean with the median and read the skew and the excess kurtosis (0 for a normal distribution). All four are close to symmetric: the largest skew is 0.17 (BMI), far below the rule-of-thumb threshold of 1, and each mean lies within one unit of its median. Physical activity has only 772 values, because some people did not report it. Income is an ordered categorical variable: 50 to 75k is the most common band (207 people), and 35 people did not report income.

round(prop.table(table(phaa$gender, phaa$smoker), margin = 1) * 100, 1)
tab <- table(Smoker = phaa$smoker, Hypertension = phaa$hypertension)
tab
(tab["Yes", "Yes"] / tab["Yes", "No"]) / (tab["No", "Yes"] / tab["No", "No"])
Console output
No Yes Man 83.6 16.4 Non-binary 88.9 11.1 Woman 82.1 17.9 Hypertension Smoker No Yes No 649 15 Yes 128 8 [1] 2.704167

With margin = 1 the percentages run along the rows, so each row adds to 100% and shows the share of smokers within each gender: 17.9% of women, 16.4% of men and 11.1% of the 27 non-binary participants (a small, unstable row). In the 2×2 table, 8 smokers and 15 non-smokers have hypertension. Line 22 divides the odds of hypertension among smokers by the odds among non-smokers, both taken from the cells of the table, and prints the odds ratio, 2.70: the odds of hypertension are about 2.7 times as high among smokers. This descriptive odds ratio rests on very few people with hypertension and makes no allowance for chance or for other differences between the groups.

par(mfrow = c(2, 2))
hist(phaa$systolic_bp, main = "Systolic BP", xlab = "mmHg")
boxplot(systolic_bp ~ smoker, data = phaa, ylab = "Systolic BP (mmHg)")
barplot(table(phaa$gender), main = "Gender")
plot(phaa$age, phaa$systolic_bp, xlab = "Age (years)", ylab = "Systolic BP (mmHg)")
par(mfrow = c(1, 1))
Four plots of the course data. Top left, a histogram of systolic blood pressure with one peak near 110 mmHg. Top right, boxplots of systolic blood pressure for non-smokers and smokers: the smokers' median (114) is above the non-smokers' median (108), and three non-smoker readings of 140 to 142 lie above the upper whisker. Bottom left, a bar chart of gender: 359 men, 27 non-binary people and 414 women. Bottom right, a scatter plot of systolic blood pressure against age, rising from lower left to upper right.

The histogram shows one continuous variable with a single peak near 110 mmHg. The boxplots show a higher median for smokers (114 against 108 mmHg); three non-smoker readings of 140 to 142 mmHg lie beyond the upper whisker, and the smokers have none. The bar chart is for a categorical variable, and the scatter plot shows systolic BP rising with age with no point far from the cloud.

vars <- c("age", "gender", "bmi", "systolic_bp", "phys_act_min", "income")
t1 <- CreateTableOne(vars = vars, strata = "smoker", data = phaa, test = FALSE)
print(t1, smd = TRUE, missing = TRUE)
Console output
Stratified by smoker No Yes SMD Missing n 664 136 age (mean (SD)) 45.11 (13.35) 45.35 (14.90) 0.017 0.2 gender (%) 0.098 0.0 Man 300 (45.2) 59 (43.4) Non-binary 24 ( 3.6) 3 ( 2.2) Woman 340 (51.2) 74 (54.4) bmi (mean (SD)) 21.03 (3.24) 19.47 (2.94) 0.504 0.1 systolic_bp (mean (SD)) 107.16 (11.00) 113.45 (11.59) 0.557 0.2 phys_act_min (mean (SD)) 176.05 (83.43) 176.09 (77.12) 0.001 3.5 income (%) 0.118 4.4 <25k 76 (12.0) 18 (13.5) 25-50k 135 (21.4) 24 (18.0) 50-75k 168 (26.6) 39 (29.3) 75-100k 139 (22.0) 31 (23.3) >100k 114 (18.0) 21 (15.8)

The columns are the exposure groups, non-smokers (664) and smokers (136); smoking defines the columns and is not also a row. Continuous rows show mean (SD) and categorical rows n (%) in each level; print(t1, nonnormal = "...") would show a skewed variable as median [IQR]. The Missing column gives the percentage missing (3.5% for physical activity, 4.4% for income). test = FALSE leaves out p-values; the SMD column gives standardised differences, where values below about 0.1 are read as a negligible imbalance. BMI (0.504) and systolic BP (0.557) differ by about half a standard deviation.

dep_items <- phaa[, paste0("dep", 1:7)]      # dep1 to dep7
anx_items <- phaa[, paste0("anx", 1:5)]      # anx1 to anx5
dep_alpha <- alpha(dep_items)
round(dep_alpha$total$raw_alpha, 3)          # alpha, depression items
round(cbind(dep_alpha$alpha.drop["raw_alpha"], dep_alpha$item.stats["r.drop"]), 3)
round(alpha(anx_items)$total$raw_alpha, 3)   # alpha, anxiety items
fa.parallel(dep_items, fa = "fa")            # how many factors?
factanal(na.omit(dep_items), factors = 1)
combined <- na.omit(cbind(dep_items, anx_items))   # complete rows only
factanal(combined, factors = 2, rotation = "varimax")
Console output
[1] 0.904 raw_alpha r.drop dep1 0.889 0.723 dep2 0.886 0.744 dep3 0.888 0.731 dep4 0.887 0.741 dep5 0.892 0.690 dep6 0.889 0.717 dep7 0.896 0.654 [1] 0.855 Parallel analysis suggests that the number of factors = 1 and the number of components = NA Call: factanal(x = na.omit(dep_items), factors = 1) Uniquenesses: dep1 dep2 dep3 dep4 dep5 dep6 dep7 0.405 0.368 0.401 0.391 0.465 0.417 0.523 Loadings: Factor1 dep1 0.772 dep2 0.795 dep3 0.774 dep4 0.781 dep5 0.731 dep6 0.763 dep7 0.691 Factor1 SS loadings 4.030 Proportion Var 0.576 Test of the hypothesis that 1 factor is sufficient. The chi square statistic is 12.7 on 14 degrees of freedom. The p-value is 0.551 Call: factanal(x = combined, factors = 2, rotation = "varimax") Uniquenesses: dep1 dep2 dep3 dep4 dep5 dep6 dep7 anx1 anx2 anx3 anx4 anx5 0.409 0.373 0.398 0.390 0.466 0.414 0.521 0.416 0.475 0.430 0.473 0.498 Loadings: Factor1 Factor2 dep1 0.748 0.177 dep2 0.777 0.150 dep3 0.748 0.205 dep4 0.757 0.192 dep5 0.717 0.143 dep6 0.751 0.148 dep7 0.682 0.115 anx1 0.141 0.751 anx2 0.124 0.714 anx3 0.111 0.747 anx4 0.117 0.716 anx5 0.139 0.695 Factor1 Factor2 SS loadings 3.921 2.815 Proportion Var 0.327 0.235 Cumulative Var 0.327 0.561 Test of the hypothesis that 2 factors are sufficient. The chi square statistic is 36.88 on 43 degrees of freedom. The p-value is 0.733
Parallel analysis scree plot for the seven depression items. The blue line for the actual data starts at an eigenvalue of about 4 for factor 1 and drops to near 0 for factors 2 to 7. The red lines for simulated and resampled data start near 0.5 to 0.7 and fall toward 0. Factor 1 lies far above the red lines and factor 2 falls below them.

Cronbach's α for the seven depression items is 0.904 (0.80 or higher is good), and for the five anxiety items 0.855. Every alpha-if-dropped value is below 0.904, so every item adds to the scale; dep7 contributes least (0.896 if dropped, item-rest correlation 0.654). Parallel analysis keeps one factor, all seven one-factor loadings lie between 0.691 and 0.795 (well above 0.40), the factor explains 57.6% of the item variance, and the test that one factor is enough gives p = 0.551. In the two-factor solution every depression item loads on Factor 1 and every anxiety item on Factor 2 (own loadings at least 0.68, cross-loadings at most 0.21), so the two scales measure distinct, though related, constructs. na.omit() keeps the complete rows, so the 15 people who skipped dep4 are left out of these checks.

n_dep <- rowSums(!is.na(dep_items))          # items answered by each person
phaa$dep_score <- ifelse(n_dep >= 6, rowMeans(dep_items, na.rm = TRUE) * 7, NA)
n_anx <- rowSums(!is.na(anx_items))
phaa$anx_score <- ifelse(n_anx >= 4, rowMeans(anx_items, na.rm = TRUE) * 5, NA)
summary(phaa$dep_score)
Console output
Min. 1st Qu. Median Mean 3rd Qu. Max. 7 14 21 21 27 35

Each score is the mean of the answered items multiplied by the number of items (a prorated sum), calculated only for people who answered all but at most one item. A skipped item is never counted as 0: a simple row sum would understate the scores of the 15 people who skipped dep4 by 1.8 to 4.5 points (2.8 on average).

qqnorm(phaa$systolic_bp); qqline(phaa$systolic_bp)
shapiro.test(phaa$systolic_bp)
Console output
Shapiro-Wilk normality test data: phaa$systolic_bp W = 0.99258, p-value = 0.0005134
Normal Q-Q plot of systolic blood pressure. The points follow the reference line closely from about 90 to 140 mmHg; at the lower left the 18 lowest readings, all 85 mmHg, form a short flat run below the line.

The Q-Q plot follows the reference line closely except at the lower end, where the 18 lowest readings are all exactly 85 mmHg. The Shapiro-Wilk test has normality as its null hypothesis, so p = 0.0005 means that a departure was detected; with 798 readings the test detects even small departures, while the skew of 0.05 shows that the distribution is close to symmetric. The decision rests on the plot and the skew together.

Files for this activity Data Code Answer key

R Reflect on what you just ran

Answer each question from your own output, as on the midterm exam: choose and justify the summaries, set out the table, judge the check.

1. Lines 14 to 16 describe age, BMI, systolic blood pressure and minutes of physical activity a week, and the distribution of household income. For income, type round(100 * prop.table(table(phaa$income)), 1) at the end of the script and run it to print the percentage in each band. For age, systolic BP, physical activity and income, state the measure of central tendency and the measure of spread you would report, and justify each choice with evidence from the output. Then state what the skew and kurtosis of systolic BP indicate about the shape of its distribution and its tails. (describe() reports excess kurtosis, which is 0 for a normal distribution.)

Model answerAge: report the mean and standard deviation, 45.15 (13.62) years, because the skew of 0.12 is far below the rule-of-thumb threshold of 1, the excess kurtosis of −0.36 is close to 0 and the mean lies close to the median of 45, so the distribution is close to symmetric. Systolic BP: mean and SD, 108.22 (11.34) mmHg, because its skew (0.05) and excess kurtosis (−0.31) are both close to 0 and the mean and median (109) agree. Physical activity: mean and SD, 176.05 (82.37) minutes a week, with the missing values reported (only 772 people have a value), because its skew (0.16) and excess kurtosis (−0.01) are close to 0 and the mean and median (176.5) agree. Household income: the count and percentage in each band among the people with income data, read from line 16 and your prop.table() line (under 25k 94, or 12.3%; 25 to 50k 159, or 20.8%; 50 to 75k 207, or 27.1%; 75 to 100k 170, or 22.2%; over 100k 135, or 17.6%), with the modal band (50 to 75k) as the typical value and the 35 missing values reported. Income is recorded in ordered categories, so a mean and SD of the bands have no meaning; the spread is shown by the percentage distribution, and because the bands are ordered the median category, 50 to 75k, can also be given. For systolic BP, a skew close to 0 indicates a symmetric distribution, and the small negative excess kurtosis indicates slightly lighter tails and a slightly flatter peak than a normal distribution (mildly platykurtic), so the distribution is close to normal: a single, nearly symmetric hump centred near 108 mmHg, with the mean and median at the same point. A skew above about 1, a mean well above the median, or an SD larger than the mean for a variable that cannot fall below zero would instead call for the median and IQR, because a few extreme values pull the mean and SD towards the long tail.

2. Using the output of line 35, set out the structure of a Table 1 for this sample stratified by smoking status: say what the columns are, list the rows you would include and the statistic you would report in each cell, and say how missing values are reported. Then use the SMD column to say which characteristics are balanced between smokers and non-smokers and which differ.

Model answerColumns: the exposure groups, non-smokers (n = 664) and smokers (n = 136), each headed by its n, with an overall column (n = 800) if wanted; smoking defines the columns, so it is not also a row. Rows: the characteristics that matter for the analysis, here age, gender, BMI, physical activity and household income, with education and any other potential confounders added the same way, and systolic BP (the main outcome) as a descriptive row if wanted. Statistic for continuous rows: mean (SD) for approximately normal variables, as for age, 45.1 (13.4) against 45.3 (14.9) years, and for BMI, systolic BP and physical activity, which are all close to symmetric; a skewed variable would be shown as median (IQR), which print(t1, nonnormal = ...) produces. Statistic for categorical rows: n (%) in each category, for example 340 (51.2%) and 74 (54.4%) women, with the number missing reported for every variable with gaps (income is missing for 35 people, 4.4%, and physical activity for 28, 3.5%). For the comparison column report the standardised difference (SMD) rather than a p-value, as the STROBE guideline advises, because a p-value depends heavily on sample size; an SMD below about 0.1 is read as a negligible imbalance. Age (0.017), physical activity (0.001) and gender (0.098) are balanced and income (0.118) nearly so, while BMI (0.504; 19.5 against 21.0 kg/m²) and systolic BP (0.557; 113.5 against 107.2 mmHg) differ by about half a standard deviation. BMI may be a mediator, because smoking can lower body weight, so whether a later model adjusts for it is decided from subject knowledge about causal order, not from the table.

3. Run your own code: at the end of the script, type shapiro.test(phaa$diastolic_bp) and run it. A colleague notes that shapiro.test(phaa$systolic_bp) gives p = 0.0005 and says that systolic BP must therefore be reported as median (IQR), whereas your output shows that diastolic BP is normal. Do you agree? Explain, using the Q-Q plot from line 57, the skew from line 15 and your own output.

Model answerYour line gives W = 0.99672 and p = 0.09998 for diastolic BP. Disagree: the Shapiro-Wilk test has normality as its null hypothesis, so p = 0.0005 for systolic BP says only that some departure from normality was detected. With 798 readings the test has the power to detect departures too small to matter, so the size of the departure is judged from the Q-Q plot and the skew. The Q-Q plot follows the line closely except for a short flat run of 18 readings at 85 mmHg at the lower end, the skew is 0.05 and the excess kurtosis −0.31, and the mean (108.22) and median (109) almost agree, so systolic BP is close to normal. Report it as mean (SD), 108.2 (11.3) mmHg; parametric methods such as a t-test are reasonable, and nothing calls for a transformation or a rank-based method. For diastolic BP, p = 0.10 means only that no departure was detected, not that normality has been proven; its skew (0.07) and excess kurtosis (0.05), shown in Worked code 2.11, lead to the same decision, mean (SD), 70.2 (9.1) mmHg. The decision rests on the plot and the skew together, not on the p-value alone.
Saved.

Activity 2.3 turns the cleaned course data into the summaries, tables and plots that a report opens with, and its questions ask you to justify each choice from your own output. This section has described the measures of centre, spread and shape, frequency tables and the odds ratio, the construction of a stratified Table 1, the choice of plots and the assessment of normality, with the construction of scale scores as background. The key takeaways below summarise each of these points, and the knowledge check that follows reviews them.

Key Takeaways

  • 5.2 The right summary depends on both the type of variable and the shape of its distribution. For a roughly symmetric continuous variable, report the mean and SD; for a skewed one, report the median and IQR, because a few extreme values pull the mean towards the long tail and inflate the SD; and for categories, report n (%) in each level, with the mode (and, for ordered levels, the median category) as the typical value, never a mean of the codes. To justify your choice from describe() output, cite the evidence: |skew| above about 1 signals substantial skew, excess kurtosis is 0 for a normal distribution (positive means heavier tails, or leptokurtic; negative means lighter tails, or platykurtic), and a mean well above the median signals a long right tail, as does an SD larger than the mean for a variable that cannot fall below zero. When you sketch the shape the output implies, draw a right-skewed variable as a tall peak at low values with a tail thinning out to the right and the median marked to the left of the mean, and a near-normal variable as a single symmetric hump with the mean and median at the same point. Report the number missing alongside every summary, as the worked example on length of stay, quality of life and education does.
  • 6.1 A frequency table gives the count and percentage in each category, and a cross-tabulation shows two categorical variables together so that you can compare groups. In R, table() gives the counts and prop.table(table(x, y), margin = 1) turns them into row proportions, so each row sums to 1 (100%); check which way the percentages run before you interpret them. R gives the odds and the odds ratio from a 2×2 table in a line or two, as in Worked code 2.12, which prints an OR of 2.70 for hypertension by smoking status in the course data. The formula explains what that number is: with a = exposed cases, b = exposed controls (non-cases), c = unexposed cases and d = unexposed controls, OR = (a × d) / (b × c), the odds of exposure among cases divided by the odds among controls (in Worked Example 2.9, an odds ratio of 6.0). An OR of 1 means no association, above 1 means the exposure is more common (in odds terms) among cases and below 1 means it is less common, and a descriptive OR makes no allowance for chance or for other differences between the groups. The most common slip is reading the wrong cells, so check that the rows and columns of the R table match the a, b, c, d layout before you interpret the odds ratio it gives.
  • 6.2 A Table 1 describes the people in your study before any model is fitted, and stratifying it by exposure shows whether potential confounders are balanced between the groups. Setting it out takes three decisions. The columns are the exposure groups, often with an overall column, each headed by its n, and the variable that defines the columns does not also appear as a row. The rows are the characteristics that matter for the analysis (demographics, potential confounders and other covariates, and the outcome if wanted), and each cell holds the statistic that suits that row: mean (SD) for approximately normal variables, median (IQR) for skewed ones and n (%) for each level of a categorical variable, with the number missing reported for every variable that has gaps. For the comparison column this course follows the STROBE guideline and reports standardised differences rather than p-values, because a p-value depends heavily on sample size; a standardised difference below about 0.1 is commonly read as a negligible imbalance.
  • 7.1 Choose a plot by asking what kind of variable you have and what question the picture should answer. A histogram shows the shape of one continuous variable (peaks, tails, gaps and stray clusters, such as readings with a shifted decimal point); try two or three bin numbers, because too few bins can merge two peaks and too many turn the outline into noise. Side-by-side boxplots, such as boxplot(systolic_bp ~ smoker, data = phaa), compare a continuous variable across groups and show the median, the IQR and the points beyond the 1.5 × IQR fences. Bar charts are for categorical variables only, never for continuous ones; scatter plots show how two continuous variables relate and can reveal a bivariate outlier, such as 191 cm with 46 kg, a combination that is implausible even though each value looks ordinary on its own; and line graphs show trends over time. Plot before you fit any model, because a picture shows in seconds whether a comparison will be driven by a few extreme values or by a real shift in the centre.
  • 7.2 Before choosing between parametric methods (the t-test, ANOVA, Pearson correlation) and their rank-based alternatives, check whether each continuous variable is close enough to normal. Look at a Q-Q plot (qqnorm() followed by qqline()), where points along the line suggest normality, points curving away at one end suggest skew and points bending away at both ends in an S shape suggest heavy tails, and look at the histogram and the skew value beside it. A formal test such as shapiro.test() tests the null hypothesis that the data are normal, so a small p-value means a departure was detected, but in a large sample it flags departures too small to matter: systolic BP in the course data gave p = 0.0005 with a skew of only 0.05. Decide from the plot and the skew together: if the plot is close to a straight line and |skew| is below about 1, use mean (SD) and parametric methods; otherwise summarise with median (IQR) and either transform the variable or use a rank-based method (Wilcoxon rank-sum, Kruskal-Wallis or Spearman correlation).
  • For context: before questionnaire items are combined into a scale score, two checks are needed. Cronbach's α asks whether the items hang together (about 0.70 or higher is acceptable and 0.80 or higher good, as for the seven depression items, with α = 0.904), and factor analysis asks whether they measure one underlying construct (parallel analysis suggests how many factors to keep, and loadings of about 0.40 or more are meaningful). Reverse-coded items must be flipped before either check, and a skipped item should never be counted as 0 when the score is built, because that understates the person's score.
Knowledge check: this section

1. A survey of 905 commuters produced the output below for daily commute time in minutes and the Perceived Stress Scale score (possible range 0 to 40). Which choice of summaries and description of shape is correct?

> describe(dat[, c("commute_min", "pss")])
              n  mean    sd median min max  skew kurtosis
commute_min 905 31.60 24.80   24.0   0 165  1.83     4.10
pss         890 18.20  6.10   18.0   2  36  0.05    -0.29
A large sample does not make a skewed variable symmetric, so commute time still needs the median (IQR). For the stress score, an excess kurtosis near 0 means tails close to normal, not a reason for the median.
Choosing the median for commute time is right, but positive skew means a long right tail with the mean above the median, not a left tail. The reason given reverses the direction of skew.
Commute time has skew above 1, a mean well above the median and heavy tails, so median (IQR) fits. The stress score has skew and excess kurtosis near 0 and agreeing mean and median, so mean (SD), with missing values noted.
describe() printing an SD for every variable does not make it the right summary. The choice follows each variable's shape, and commute time is strongly right-skewed.
The answer is (c). For each variable, read the skew, the excess kurtosis and the gap between the mean and the median. Commute time has a skew of 1.83 (above the rule-of-thumb threshold of 1), a mean of 31.6 against a median of 24.0, and an excess kurtosis of 4.10 (heavier tails than normal, leptokurtic), so its centre and spread are the median and IQR. The stress score has a skew of 0.05, an excess kurtosis of −0.29 (slightly lighter tails) and a mean of 18.2 against a median of 18.0, so it is close to normal and is reported as mean (SD), with the 905 − 890 = 15 missing values. Option (b) reverses the meaning of positive skew, which is a long right tail with the mean above the median, and option (a) treats a small negative excess kurtosis as a reason for the median, when a value near 0 means tails close to those of a normal distribution. (Section 3, 5.2.3 Measures of Shape and Worked Example 2.7; Objective 5.2)

2. A case-control study of a gastrointestinal outbreak at a summer camp asks whether eating from the salad bar was associated with illness. The analyst enters the counts in R and obtains the output below. Which reading of the output is correct?

> salad <- matrix(c(36, 24, 30, 90), nrow = 2,
+   dimnames = list(Salad = c("Ate", "Did not eat"), Group = c("Case", "Control")))
> salad
             Group
Salad         Case Control
  Ate           36      30
  Did not eat   24      90
> odds_cases    <- salad["Ate", "Case"] / salad["Did not eat", "Case"]
> odds_controls <- salad["Ate", "Control"] / salad["Did not eat", "Control"]
> round(c(cases = odds_cases, controls = odds_controls, OR = odds_cases / odds_controls), 3)
   cases controls       OR
   1.500    0.333    4.500 
The output gives the odds of exposure in cases and controls and their ratio. An odds ratio above 1 means exposure is more common, in odds terms, among cases, and 4.5 is a strong positive association.
Dividing the proportions exposed compares proportions, not odds, so the figure is not an odds ratio. The odds ratio is the ratio of the two odds that R prints.
The value for controls is the odds of exposure in that group, not the odds ratio. Reading one group's odds as the ratio gives the wrong direction and size of association.
The value for cases is the odds of exposure among cases, not the odds ratio. The ratio compares the cases' odds with the controls' odds.
The answer is (a). The output gives the odds of exposure among cases (36 / 24 = 1.5), the odds among controls (30 / 90 = 0.333) and their ratio, the odds ratio of 4.5. An OR above 1 means the exposure is more common, in odds terms, among cases, and 4.5 is a strong positive association; like any descriptive OR it makes no allowance for chance or for other differences between the groups. Option (b) is the tempting one: 0.60 / 0.25 = 2.4 compares the proportions exposed, not the odds, so it is not an odds ratio. Options (c) and (d) read the odds in one group as if it were the odds ratio. (Section 3, 6.1 Frequency Tables, Cross-Tabulations and the Odds Ratio, Worked Examples 2.8 and 2.9 and Worked code 2.12; Objective 6.1)

3. An analyst plans four displays for a health survey: (i) the shape of HbA1c values in the whole sample, (ii) BMI compared across four neighbourhoods, (iii) the percentage of respondents in each smoking category, and (iv) whether systolic blood pressure rises with age. Which set of plots matches the variables and the questions?

Each display is mismatched: a bar chart is for categorical frequencies, a scatter plot needs two continuous variables, a histogram shows one continuous variable's shape, and a line graph is for trends over time.
A single histogram of all BMI values loses the neighbourhood grouping, a scatter plot cannot show category percentages, and a bar chart of means hides the relationship between two continuous variables.
A 3D bar chart of means distorts proportions and hides the spread and outliers a group comparison should show, and a line graph is for trends over time, not two continuous variables measured once.
Each plot matches the variable type and question: a histogram for one continuous variable's shape, side-by-side boxplots for a continuous variable across groups, a bar chart for category frequencies, and a scatter plot for two continuous variables.
The answer is (d). Match each display to the variable type and the question: the shape of one continuous variable needs a histogram; a continuous variable compared across groups needs side-by-side boxplots, which show each group's median, IQR and outlying points; the frequencies of a categorical variable need a bar chart; and the relationship between two continuous variables needs a scatter plot. Option (c) is the closest distractor, but a 3D bar chart of means distorts proportions and hides the spread and outliers that a group comparison should show, and a line graph is for trends over time rather than for two continuous variables measured once. (Section 3, 7.1 Choosing a Visualisation; Objective 7.1)

4. A cohort study compares 420 night-shift workers with 610 day-shift workers. The planned characteristics are age (skew 0.3), alcoholic drinks per week (skew 2.6, with many zeros; missing for 37 people), sex, education (four ordered levels) and current smoking. Which layout for Table 1 follows the conventions taught in this lesson?

Exposure defines the columns, so it is not repeated as a row; a strongly skewed variable needs median (IQR) rather than mean (SD); and missing values are reported for each variable, not hidden.
Columns are the exposure groups with their n; each row's statistic follows type and shape, mean (SD) for near-symmetric age, median (IQR) for skewed drinks, n (%) for categories; and the number missing is reported.
Median (IQR) for drinks is right, but averaging category codes has no meaning even for ordered levels, because the codes have no fixed distance between them. Each education level is reported as n (%).
Stratifying by exposure does not bias the comparison; it is the point of Table 1, because it shows whether potential confounders are balanced between the groups.
The answer is (b). Three decisions set out a Table 1: the columns are the exposure groups, each headed by its n; the rows are the characteristics, with the statistic chosen from each variable's type and shape (age is close to symmetric, so mean (SD); drinks per week are strongly skewed, so median (IQR); each level of a categorical variable as n (%)); and the number missing is reported for every variable with gaps. Option (a) repeats the exposure as a row, summarises a skewed variable with its mean and hides the 37 missing values, and option (c) averages category codes, which has no meaning even for ordered levels. Stratifying by exposure, which (d) rejects, is the point of the table, because it shows whether potential confounders are balanced between the groups. (Section 3, 6.2.1 Setting Out a Table 1 and 6.2.2 The Comparison Column: p-Values and Standardised Differences; Objective 6.2)

5. In a survey of 4,600 adults, diastolic blood pressure has a skew of 0.06, and its normal Q-Q plot follows the reference line closely along its whole length. shapiro.test() gives p = 0.002. What should the analyst conclude?

A small p-value in a large sample can reflect a trivial departure. The straight Q-Q plot and skew near 0 show the departure is negligible, so no transformation is needed.
This reverses the null hypothesis. The Shapiro-Wilk test assumes normality, so a small p-value is evidence of some departure, not proof of normality.
With thousands of values the test can detect trivial departures. The Q-Q plot and skew show how large the departure is, here negligible, so mean (SD) and parametric methods remain reasonable.
A formal test is not more informative than a plot in a large sample, because it flags departures too small to matter. The Q-Q plot shows whether the departure is large enough to act on.
The answer is (c). Weigh three pieces of information together: the Shapiro-Wilk null hypothesis is normality, so p = 0.002 is evidence of some departure; the sample is large, so the test has the power to detect trivial departures; and the Q-Q plot and the skew show how large the departure is, here negligible. Option (a) is tempting because p is small, but a significant test in a large sample does not by itself call for a transformation or a rank-based method, and option (b) reverses the null hypothesis. (Section 3, 7.2.1 Visual Checks and Formal Tests and 7.2.2 From the Checks to a Decision; Objective 7.2)

6. A cross-sectional survey of 480 adults in a rural health region asks whether household food insecurity is associated with type 2 diabetes. The output below describes age and the distance from home to the nearest grocery store (dist_km), and cross-tabulates diabetes by food security. Which set of Table 1 entries, stratified by food security, follows the conventions taught in this section?

> describe(svy[, c("age", "dist_km")])
           n  mean    sd median  min  max skew kurtosis
age      480 47.80 13.20   47.0 19.0 84.0 0.18    -0.40
dist_km  466  6.90  8.40    3.9  0.2 52.5 2.64     8.31
> quantile(svy$dist_km, c(0.25, 0.75), na.rm = TRUE)
25% 75%
1.8 8.6
> sum(is.na(svy$dist_km))
[1] 14
> tab <- table(svy$food_insec, svy$diabetes)
> tab

            no yes
  secure   288  32
  insecure 120  40
> rowSums(tab)
  secure insecure
     320      160
> round(100 * prop.table(tab, margin = 1), 1)

           no yes
  secure   90  10
  insecure 75  25
describe() printing a median does not decide the summary. Age is near symmetric, so mean (SD), while distance is strongly right-skewed, so median (IQR); the exposure also defines the columns and is not a row.
Age is right, but a large sample does not make a skewed variable symmetric, so distance needs median (IQR). Diabetes is categorical and is reported as n (%), not as the mean of its codes.
Placing exposure groups side by side does not bias the comparison; it shows whether characteristics are balanced. A skewed variable and a categorical one also should not be summarised as mean (SD).
Columns are the exposure groups with their n, and each row follows the variable's type and shape: mean (SD) for near-symmetric age, median (IQR) with missing values for skewed distance, and n (%) for diabetes.
The answer is (d). Three decisions set out the table. The columns are the exposure groups, each headed by its n (320 and 160 from rowSums()), so food security is not also a row. The statistic in each row follows the variable's type and shape: age is close to symmetric (skew 0.18, excess kurtosis −0.40, mean 47.8 against median 47.0), so mean (SD); distance is strongly right-skewed (skew 2.64, excess kurtosis 8.31, mean 6.9 against median 3.9 km, and an SD larger than the mean), so median (IQR), with the 14 missing values reported; and diabetes is categorical, so n (%) from the row percentages, 10% and 25%. Option (b) is tempting because the sample is large, but a large sample does not make a skewed variable symmetric, and a categorical variable is reported as n (%); option (a) reverses the two continuous rows and repeats the exposure as a row; and option (c) gives up the stratification that shows whether characteristics are balanced between the groups. (Section 3, 5.2.2 Measures of Spread, reporting convention, Worked Example 2.7, 6.1 Frequency Tables, Cross-Tabulations and the Odds Ratio and 6.2.1 Setting Out a Table 1; Objectives 5.2 and 6.2)

● Complete the knowledge check to continue.

Section 4

Final Assessment

⏱ Estimated time: 15 minutes

Bringing It All Together

This lesson followed a single dataset from raw to descriptive. The section on data quality named the errors (collection, entry, storage, and analysis-stage problems) and unpacked the missing-data mechanisms (MCAR, MAR, MNAR) that determine which cleaning approaches are valid. The section on cleaning strategies turned that vocabulary into a workflow: systematic range, type, and consistency checks; outlier triage; principled missing-data handling; transformations and derived variables; and a documentation trail that a peer reviewer could follow. The section on descriptive analyses closed the loop with the layer that every paper opens with: central tendency, spread, shape, frequency tables, visualisations, scale checks, and a Table 1 that stratifies on the primary exposure or outcome.

Descriptive analysis is a core stage of any analysis. It is the layer where data problems surface, where assumptions get tested, and where readers calibrate how much trust to place in the inferential models that follow. Regression models can then be fitted knowing that the dataset under them has been examined and understood.

Key Takeaways from this lesson

  1. You can now read a few raw rows beside a data dictionary, name the errors and coding faults you see (transcription, coding, logical inconsistency, out-of-range and duplicate errors; missing codes that could be real values, compound codes, text in numeric fields, mixed date formats and inconsistent missing values), say where in the pipeline each one entered, and state how to correct it. For each fault, give both halves of the answer, the problem and its best-practice correction, and remember that every correction is made in a script that reads the raw file, never by editing the raw file by hand.
  2. You can classify a described pattern of missing data as MCAR, MAR or MNAR by asking what the missingness depends on: nothing about the people, observed variables only, or the missing value itself. Answer in three parts every time: the label, a one-sentence justification that names what the missingness depends on, and the consequence of a complete-case analysis (unbiased but less precise under MCAR, possibly biased under MAR, and biased under MNAR in a way that imputation under the MAR assumption does not correct).
  3. You can run range, type, consistency and outlier checks, decide whether each flagged value should be kept, transformed or excluded, and write the rule in a cleaning log. In a question built on R output, compare the Min. and Max. from summary() with the documented valid range, set impossible or miscoded values to NA while keeping the row, and never choose a transformation to fix an error; then write a rule that names the variable, the limit broken, the action, the number of values affected, and who made the change and when.
  4. You can choose a missing-data method that suits the plausible mechanism and the amount missing: complete-case analysis when little is missing and MCAR is plausible, and multiple imputation pooled with Rubin's rules under a stated MAR assumption otherwise. Remember that single imputation treats guessed values as known, which makes standard errors too small, and that the share of rows an analysis loses can be much larger than the share missing on any one variable.
  5. You can choose, obtain in R and interpret the measures of centre, spread and shape that suit each variable's type and distribution, and sketch the shape a summary implies. When you justify a choice from describe() output, cite the evidence: |skew| above about 1, excess kurtosis well away from 0 and a mean far from the median point to the median (IQR); skew and excess kurtosis near 0 with the mean close to the median point to the mean (SD); and categorical variables are summarised as n (%) with the number missing reported.
  6. You can build frequency tables and cross-tabulations, obtain in R and interpret an odds ratio from a 2×2 table, and set out a Table 1 stratified by exposure. Check which cells of the R table are a, b, c and d, the cells in OR = (a × d) / (b × c), before you interpret the odds ratio that R prints, and in a Table 1 head each exposure column with its n, choose the statistic row by row, report the missing values, and leave the exposure itself out of the rows.
  7. You can choose a plot that suits the variable and the question, and judge whether a variable is close enough to normal for parametric methods. Base that judgement on a Q-Q plot, a histogram and the skew together rather than on a Shapiro-Wilk p-value alone, because in large samples the test flags departures too small to matter; when the shape is clearly non-normal, transform the variable or use a rank-based method.

Final Knowledge Assessment

You must answer all 16 questions correctly (100%) to finish this lesson.

Final assessment: the 16 questions

1. One participant's record shows smoking status "Never smoked" and cigarettes per day = 15. Each value lies within its valid range in the data dictionary. What type of error is this, and which check would find it?

Each value lies within its valid range, so a range check on cigarettes per day would pass it. The problem is between two fields, not within one.
Two values each valid on their own contradict each other, which is a logical inconsistency. A consistency check comparing fields within the record finds it, and the source shows which field is wrong.
A duplicate record is the same observation entered more than once. Here a single record contains contradictory fields, which duplicated() cannot detect.
This is a specific contradiction in one record, not random variation, and the lesson does not treat errors as something that can be ignored. It needs a consistency check and a look at the source.
The answer is (b). The key information is that each value is valid on its own but the two contradict each other: a person who has never smoked cannot smoke 15 cigarettes a day. A contradiction between fields in one record is a logical inconsistency, and a cross-field consistency check finds it, after which the source is consulted to learn which field is wrong. A range check (a) examines one variable at a time, so it passes both values. (Section 1, 1.2.1 Screening, Diagnosing and Editing, Logical Inconsistencies card; Section 2, 3.1 Systematic Verification: Range, Type and Consistency Checks, Consistency Checks panel; Objectives 1.2 and 3.1)

2. A clinic sends the raw extract and data dictionary below. Which pair of corrections follows best practice?

id   n_visits  first_visit
A01  3         2024-11-05
A02  none      05/11/2024
A03  1         5 Nov 24

Data dictionary as supplied
  n_visits     number of clinic visits in 2024
  first_visit  date of first visit
"None" means zero visits, so converting it to NA turns a real answer into a missing one. Deleting rows with other date formats discards valid data instead of converting the dates.
Leaving n_visits as text stops it being analysed as a number, and retyping dates by hand leaves no record of the change. Both corrections belong in a script.
Coding "none" as 99 creates a value that could be read as 99 visits. Converting dates without checking ambiguous ones risks fixing a wrong interpretation.
Recording "none" as 0 keeps the column numeric with the right value, and converting dates to one stated format in the script, checking ambiguous ones at the source, removes the inconsistency.
The answer is (d). Two coding faults are visible: text in a numeric field ("none" in n_visits) and inconsistent, partly ambiguous date formats. "None" means zero visits, so the correct value is the number 0, which keeps the column numeric; converting it to NA (a) turns a real answer into a missing one, and coding it 99 (c) creates a code that could be read as 99 visits. Dates are stored as one date variable in one stated format, ambiguous entries are checked at the source, and every change is made in a script rather than by hand (b), so that it can be checked and repeated. (Section 1, 1.2.3 Coding Faults in a Raw File, table of coding faults; Objective 1.2)

3. In a survey of 1,200 university students, the item on alcoholic drinks per week is missing for 9%. Follow-up interviews suggest that the heaviest drinkers were the most likely to skip the item, and no recorded variable explains who skipped it. Which classification and consequence are correct?

Missingness depends on the unrecorded number of drinks itself, which is MNAR. Heavy drinkers are under-represented among complete cases, imputation under MAR cannot fix this, and sensitivity analyses are needed.
The amount missing does not determine the mechanism. Missingness is linked to the heaviest drinkers, so it is not unrelated to the participants and cannot be MCAR.
Learning the reason in interviews does not make the missingness depend only on recorded variables. No recorded variable explains who skipped, so the pattern is MNAR and imputation under MAR does not remove the bias.
The label is right, but under MNAR the complete cases lack the heaviest drinkers, so a complete-case mean is biased downward, not merely less precise.
The answer is (a). The deciding information is what the missingness depends on: the students most likely to skip the item are those with the highest values of it, and no recorded variable accounts for this, so the probability of missingness depends on the missing value itself. The heaviest drinkers are therefore under-represented among the complete cases, which pulls the mean down (as the missing high earners do in Worked Example 2.4), and multiple imputation under MAR cannot recover values that depend on what was never recorded. Option (b) confuses the amount missing with the mechanism, and option (c) fails because knowing the reason for skipping does not make the missingness depend only on recorded variables. (Section 1, 2.1 Missing-Data Mechanisms, "Classifying a described pattern"; Section 2, 4.1.3 Reasoning About the Mechanism, Worked Example 2.4; Objective 2.1)

4. Household income is missing for 12% of 3,000 survey respondents. Respondents without an income value are on average 9 years older than those with one, and age is recorded for everyone. A colleague writes: "The mechanism cannot be checked, so we will assume MCAR and analyse complete cases, which gives unbiased estimates." What is the best response?

MCAR is not a default for unverifiable mechanisms. The observed age gap shows missingness is related to an observed variable, which is evidence against MCAR.
A link with an observed variable is evidence of MAR, not proof of MNAR. The data cannot rule out MNAR, but a valid analysis under a stated MAR assumption with sensitivity analysis is still possible.
The age gap shows missingness is related to an observed variable, so MCAR is unsafe and MAR given age is more plausible. Multiple imputation using age, a stated assumption, and a sensitivity analysis for MNAR is the defensible plan.
Mean imputation gives everyone the same value regardless of age, so it neither uses the information in age nor keeps the true variability. It does not remove the bias.
The answer is (c). The observed data cannot show whether missingness depends on income itself, but they can show that it depends on an observed variable, and a 9-year age gap does exactly that, so MCAR is not a safe assumption. Because the complete cases are younger than the full sample, complete-case estimates can be biased, and multiple imputation using age, with a stated MAR assumption and a sensitivity analysis for MNAR (a known risk for income), is the defensible plan. Option (b) overreaches, because a link with an observed variable is evidence of MAR, not proof of MNAR, and option (d) gives everyone the same value whatever their age, so it neither uses the information in age nor keeps the true variability. (Section 1, 2.1 Missing-Data Mechanisms; Section 2, 4.1.3 Reasoning About the Mechanism; Objectives 2.1 and 4.1)

5. A study of 900 participants analyses four variables. Because a printing fault skipped one page in a random set of questionnaires, 14 participants (1.6%) lack a value on one of the four variables; nothing else is missing. Which approach is defensible?

Multiple imputation is not required whenever any value is missing. It is the safer choice when more is missing or when missingness is related to observed variables, neither of which applies here.
A printing fault on random questionnaires makes MCAR plausible, so complete cases are a random subset and estimates stay unbiased. With a small share missing, well below the rough guide, little precision is lost.
Narrower intervals from mean imputation are a flaw, not a benefit: single imputation treats invented values as known, which understates uncertainty.
A complete-case analysis is unbiased under MCAR regardless of the share missing; small amounts missing make it more defensible, not biased. Pairwise deletion also produces results from different subsets.
The answer is (b). The method follows from the mechanism and the amount missing: a printing fault on randomly chosen questionnaires is unrelated to the participants, so MCAR is plausible, and 14 of 900 rows (1.6%) is well below the rough 5% guide, so a complete-case analysis is unlikely to change the conclusions. Multiple imputation (a) is the safer choice when more is missing or when missingness is related to observed variables, neither of which applies here, and the narrower intervals in (c) are a flaw of single imputation, which treats invented values as if they were known. (Section 2, 4.1 Choosing a Missing-Data Strategy and 4.1.4 How Much Missing Data Is Too Much?; Objective 4.1)

6. The first cleaning pass of a cardiology survey prints the output below. The codebook gives hard limits of 30 to 220 beats per minute for resting heart rate (hr_rest). What should be done with the affected values?

> summary(dat$hr_rest)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  41.0    62.0    69.0    70.4    77.0   720.0
> sum(dat$hr_rest > 220)
[1] 2
Values beyond the hard limit are impossible, so they are errors. They are corrected from the source or set to NA with the rows kept, and the rule is recorded in the cleaning log.
The rule against deleting extreme values applies to values that are possible. A reading beyond the hard limit cannot be correct, so it is an error to exclude, not an outlier to keep.
A log transformation changes the scale of every value but leaves an impossible one in the data. The error must be corrected or set to NA.
Removing whole rows discards the participants' valid answers on every other variable. Only the impossible values are set to NA, with the rows kept.
The answer is (a). The information needed is the codebook's hard limit and the count beyond it: two readings exceed 220 beats per minute, and a resting heart rate of 720 is impossible, so these are errors rather than unusual values. Errors are excluded by correcting them from the source or setting them to NA, keeping the rows so that the participants' other answers remain (unlike d), and the rule is written in the cleaning log. Option (b) misapplies the rule about outliers, which concerns values that are possible, and option (c) fails because a transformation changes the scale of every value and leaves an impossible one in the data. (Section 2, 3.2.4 Keep, Transform or Exclude; Objectives 3.2 and 3.4)

7. Waist circumference in a sample has a mean of 92 cm and an SD of 13 cm. The analyst converts two readings, 135 cm and 60 cm, to z-scores in R. Using the z-score rule, which statement is correct?

> round((c(135, 60) - 92) / 13, 2)   # z = (x - mean) / SD
[1]  3.31 -2.46
The z-score rule judges distance in standard deviations, not in centimetres. Only a reading whose absolute z-score exceeds 3 is flagged.
The sign of a z-score does not decide whether it is flagged; the absolute value does. The 60 cm reading is within 3 SD of the mean, so it is not flagged.
The z-score is right, but a flag is a reason to investigate, not proof of an error. A waist of 135 cm is possible, so it is checked and kept if confirmed, not deleted.
R gives each reading's distance from the mean in SD units, and only an absolute value above 3 is flagged. The flagged reading is possible, so it is checked against the source and kept if confirmed.
The answer is (d). R prints each reading's distance from the mean in SD units, 3.31 for 135 cm and −2.46 for 60 cm, and only an absolute value above 3 is flagged, whatever its sign, so the 60 cm reading is ordinary (b). A flag is a reason to investigate, and a waist of 135 cm is possible, so the reading is kept if the source confirms it rather than deleted (c). Option (a) judges distance in centimetres rather than in standard deviations. (Section 2, 3.2.1 The IQR and z-Score Rules, Z-Score tab and Worked Example 2.2; Objective 3.2)

8. A colleague found 14 systolic readings of 1,100 to 1,500 mmHg, each with a shifted decimal point. She opened the raw spreadsheet, typed the corrected values (110 to 150 mmHg) over them, and saved the file under its original name. What is the main problem, and what should she have done?

Correct values are not enough: the original values are gone and nothing records what changed, why, who or when. An audit trail requires the raw file to stay untouched.
Correcting decimal-point shifts from the source forms is good practice, not guesswork. Deleting rows would discard valid data on every other variable.
The problem is that the changes cannot be checked or repeated. Confirmed corrections should be made in a script that reads the untouched raw file, logged with old and new values, and saved under a new name.
Values can be corrected when the source document confirms the true value, as with a shifted decimal point. Setting confirmed values to NA throws away correct data.
The answer is (c). The issue is not whether the new numbers are right but whether anyone can see and repeat what was done: an audit trail records what changed, why, by whom and when, and a script serves as that trail only if it runs from the raw file, which must therefore never be overwritten. Correcting decimal-point shifts from the original forms is good practice, as in the hypothetical cohort-study audit trail described in Section 2, so option (d) is wrong, and deleting the rows (b) discards valid data on every other variable. (Section 2, 3.4 Documenting the Cleaning Process; Objective 3.4)

9. Self-rated health was recorded in five ordered levels, and R gives the output below. How should the variable be summarised?

> table(dat$srh, useNA = "ifany")
     Poor      Fair      Good Very good Excellent      <NA>
       48       152       410       290       100        50
> pct <- round(100 * prop.table(table(dat$srh)), 1)   # among the 1,000 with a value
> pct
     Poor      Fair      Good Very good Excellent
      4.8      15.2      41.0      29.0      10.0
> cumsum(pct)                                          # cumulative percentages
     Poor      Fair      Good Very good Excellent
      4.8      20.0      61.0      90.0     100.0 
Self-rated health is ordered categorical, so it is summarised by the count and percentage in each level, in order, with the missing values reported. The cumulative percentages place the median, and the mode, at "Good".
Ordered levels have an order but no fixed distance between them, so the mean and SD of their codes have no meaning.
The median level is useful, but reporting it and the IQR alone hides the distribution across the five levels. A categorical variable is summarised by the count and percentage in each level.
Percentages of the recorded values are right, but the missing values must still be reported, not left out of the table.
The answer is (a). Self-rated health is an ordered categorical variable: its codes have an order but no fixed distance between them, so their mean (b) has no meaning. prop.table() gives the percentage in each level among the 1,000 recorded values, and the cumulative percentage first passes 50% at "Good" (61.0%), so the median category is "Good", which is also the most common level (the mode). Option (d) is tempting because percentages of the recorded values are correct, but the 50 missing values must still be reported. (Section 3, 5.2.4 Reading summary() and describe() Output, Worked Example 2.7; Objective 5.2)

10. Five participants report 4, 6, 8, 10 and 12 hours of moderate activity per week, and R gives the output below. Which statement reads it correctly?

> hrs <- c(4, 6, 8, 10, 12)
> mean(hrs)
[1] 8
> var(hrs)   # the variance
[1] 10
> sd(hrs)    # the standard deviation
[1] 3.162278
This reads the variance as the SD. The variance is in squared hours; the SD, its square root, gives the typical distance from the mean in hours.
The variance averages squared distances, so its units are squared hours. The SD returns to the original units and gives the typical distance of a value from the mean, which is why it is reported beside the mean.
The SD is in the original units, hours, not squared hours; it is the variance that has squared units. The SD is reported beside the mean because its units match.
A variance in squared units cannot be compared with the mean to judge skew. Skew is judged from the skew value and the gap between the mean and median.
The answer is (b). The variance averages the squared distances from the mean, so its units are squared hours; the standard deviation, its square root, returns to the original units and gives the typical distance of a value from the mean, here about 3.2 hours from a mean of 8 hours. Option (a) reads the variance as if it were the SD, option (c) gives the SD the squared units of the variance, and option (d) compares a squared quantity with the mean, whereas skew is judged from the skew value and the gap between the mean and the median. (Section 3, 5.2.2 Measures of Spread, Table 2.6 and Worked Example 2.6; Objective 5.2)

11. The output below describes age at death (years) in a regional death registry. Which reading is correct?

> describe(dat["age_death"])
             n  mean    sd median min max  skew kurtosis
age_death 2140  76.1  12.4   79.0  22 104 -1.35     2.20
A negative skew means a long left tail, not a right one. With substantial skew in either direction, the median (IQR) is reported rather than the mean (SD).
The rule-of-thumb threshold is about 1, not 2, so a skew beyond −1 is substantial. The distribution is not close to symmetric, and the mean lies below the median.
describe() reports excess kurtosis, for which 0 is the value for a normal distribution. Comparing with 3 applies only to kurtosis before 3 is subtracted, so 2.20 means heavier tails.
A negative skew beyond −1 with the mean below the median shows a long left tail, and positive excess kurtosis shows heavier tails than normal. With substantial skew, median (IQR) is reported.
The answer is (d). Three readings settle it: the skew is negative and beyond −1 (substantial skew with a long left tail, from deaths at younger ages), the mean lies below the median, and describe() reports excess kurtosis, for which 0 is the value for a normal distribution, so 2.20 means heavier tails (leptokurtic). Option (c) is the common mistake of comparing the value with 3, which applies only to kurtosis before 3 is subtracted. With substantial skew, the median and IQR describe a typical age at death better than the mean and SD. (Section 3, 5.2.3 Measures of Shape, Skewness and Kurtosis panels; Objective 5.2)

12. A case-control study of hip fracture in adults over 70 recorded regular exercise. Among 80 cases, 20 exercised regularly and 60 did not; among 160 controls, 64 exercised regularly and 96 did not. The analyst obtains the odds ratio in R. What does the output show?

> hip <- matrix(c(20, 60, 64, 96), nrow = 2,
+   dimnames = list(Exercise = c("Regular", "Not regular"), Group = c("Case", "Control")))
> hip
             Group
Exercise      Case Control
  Regular       20      64
  Not regular   60      96
> odds_cases    <- hip["Regular", "Case"] / hip["Not regular", "Case"]
> odds_controls <- hip["Regular", "Control"] / hip["Not regular", "Control"]
> round(c(cases = odds_cases, controls = odds_controls, OR = odds_cases / odds_controls), 3)
   cases controls       OR
   0.333    0.667    0.500 
This inverts the ratio, putting controls over cases. R's output shows the odds of exercise are lower among cases, so the odds ratio is below 1.
This is the proportion of cases who exercised, not an odds ratio. The odds ratio compares the odds of exposure in cases with the odds in controls.
The odds of exercise among cases are half the odds among controls. An odds ratio below 1 is an inverse association: exercise is less common, in odds terms, among people with hip fracture.
The value is right, but an odds ratio of 1 means no association. A value below 1 is an association in the inverse direction, not an absence of association.
The answer is (c). R gives the odds of regular exercise among cases (20 / 60 = 0.333) and among controls (64 / 96 = 0.667), and their ratio, 0.5. An OR of 1 means no association, so 0.5 is an association in the inverse direction rather than no association (d): regular exercise is less common, in odds terms, among people with hip fracture. Option (a) inverts the ratio (controls over cases), and option (b) is the proportion of cases who exercised (20 of 80), not an odds ratio. (Section 3, 6.1 Frequency Tables, Cross-Tabulations and the Odds Ratio, Worked Example 2.9 and Worked code 2.12; Objective 6.1)

13. A cohort study enrolled 3,000 e-cigarette users and 3,000 non-users. Its Table 1 reports age, mean (SD), as 41.2 (12.0) in users and 35.8 (11.8) in non-users (standardised difference 0.45, p < 0.001), and women as 51.5% of users and 48.5% of non-users (standardised difference 0.06, p = 0.02). Which interpretation is correct?

Standardised differences express gaps in SD units, so age shows clear imbalance and is a potential confounder, while the sex difference is below about 0.1 and negligible. The small p-value reflects the large sample.
A p-value depends heavily on sample size, so with thousands of people even a trivial difference is significant. Standardised differences show the size of the imbalance, and these two differ greatly.
Table 1 describes the sample, but that description shows whether exposure groups are balanced on potential confounders. Age is clearly imbalanced and matters for the analysis.
A variable that differs between exposure groups is a candidate confounder to adjust for if it also affects the outcome, not one to drop. Dropping it would leave any confounding uncontrolled.
The answer is (a). The standardised difference expresses each gap in standard-deviation units, so the 0.45 for age means the groups differ by almost half a standard deviation, a clear imbalance, whereas 0.06 for sex is below the usual threshold of about 0.1 for a negligible one. Option (b) is the tempting reading, but a p-value depends heavily on sample size, so with 3,000 people in each group even a 3-point difference in the percentage of women is "significant"; this is why the course reports standardised differences in Table 1. A variable that differs between exposure groups is a candidate confounder to adjust for, not to drop (d), if it also affects the outcome. (Section 3, 6.2.2 The Comparison Column: p-Values and Standardised Differences and 6.2.4 Stratification and Confounding; Objective 6.2)

14. Hospital charges were recorded for 70 patients, 35 on each of two wards. The normal Q-Q plot curves upward away from the line at the right end, the skew is 2.2 and shapiro.test() gives p < 0.001. The analyst wants to compare the two wards. Which plan is most appropriate?

Formal tests flag trivial departures only in very large samples. With 70 values, a small p-value agrees with the curved Q-Q plot and high skew, so the departure is real.
Bends at both ends in an S shape mark heavy tails, but a curve at one end marks skew. Here the curve is at the right end, so the charges are right-skewed and mean (SD) is not suitable.
Points in the middle of a Q-Q plot can lie on the line while the tails depart from it. The curve at the right end, the high skew and the small p-value all show right skew.
All three checks agree that the charges are right-skewed. A clearly skewed variable is summarised with the median (IQR) and compared with a rank-based test or after a log transformation.
The answer is (d). All three checks agree: points that curve away at the right end of a Q-Q plot show right skew, a skew of 2.2 is well above 1, and the small p-value comes from only 70 values, so it cannot be dismissed as the over-sensitivity of a very large sample (a). A clearly skewed variable is summarised with the median (IQR) and compared with a rank-based method or after a transformation. Option (b) misreads the plot, because bends at both ends in an S shape mark heavy tails, while a curve at one end marks skew. (Section 3, 7.2 Assessing Normality, Figure 2.9 and Table 2.12; Objective 7.2)

15. An analyst wants to see whether a few extreme values drive the differences in systolic blood pressure between three clinics. She plans a 3D bar chart of the three clinic means. Which display should she use instead?

Removing the third dimension fixes the distortion, but a bar of means still shows one number per clinic. It cannot show whether a few extreme values drive the difference.
Side-by-side boxplots show each clinic's median, IQR and individual points beyond the whiskers, so a real shift in centre can be told apart from a few outliers.
A single combined histogram loses the clinic grouping, so it cannot compare the clinics at all.
A line graph is for trends over time, and joining three means still shows only one number per clinic, hiding the outliers the question asks about.
The answer is (b). The question compares a continuous variable across groups and asks about extreme values, so the display must show each group's whole distribution: side-by-side boxplots draw each clinic's median, IQR and the individual points beyond the whiskers, so a real shift in the centre can be told apart from a few outliers. A bar chart of means (and its 3D version, which also distorts proportions) shows one number per clinic, a line graph (d) is for trends over time, and a combined histogram (c) loses the clinic grouping. (Section 3, 7.1 Choosing a Visualisation, Boxplots tab; Objective 7.1)

16. In a diabetes education programme with 412 participants, HbA1c at the six-month follow-up is missing for 62 (15%), almost all in the evening session (recorded in the file as session), because the laboratory closed early on evening-session days. Evening participants are younger and more often employed than daytime participants, and within each session, whether the value is missing is unrelated to the participant's HbA1c. A colleague says the values are missing completely at random, "because a laboratory closing early has nothing to do with the participants", and plans a complete-case analysis. What is the best response?

The cause is unrelated to any individual, but it affected only the evening session, whose participants differ from daytime ones. Missingness is therefore related to the people through session, so it is not MCAR.
The variable that is missing does not determine the mechanism. Within each session missingness is unrelated to HbA1c, so it is explained by a recorded variable, which is MAR, and valid analysis is possible.
Missingness depends on the recorded session and not on HbA1c within session, which is MAR. Complete cases over-represent daytime participants, so multiple imputation with session and the analysis variables, a stated assumption and a sensitivity analysis is the plan.
Mean imputation treats one filled-in value as known, which understates uncertainty, and it ignores the information in session. It does not remove the bias.
The answer is (c). Ask what the missingness depends on. The laboratory closure has nothing to do with any individual, but it affected only the evening session, and evening participants differ from daytime ones, so the missingness is related to the people through a recorded variable, session, and within each session it is unrelated to HbA1c. That is MAR rather than MCAR (a): the complete cases are mostly daytime participants, so a complete-case analysis can be biased, and losing 15% of the sample is well above the rough guide of about 5%. Multiple imputation that includes session, the outcome and the other analysis variables gives valid estimates under the stated MAR assumption, with the results pooled by Rubin's rules, and a sensitivity analysis checks the conclusion. Option (b) confuses the variable that is missing with the mechanism, and option (d) treats a single imputed value as if it were known, which understates the uncertainty and ignores the information in session. (Section 1, 2.1 Missing-Data Mechanisms, "Classifying a described pattern"; Section 2, 4.1 Choosing a Missing-Data Strategy and 4.1.4 How Much Missing Data Is Too Much?; Objectives 2.1 and 4.1)

✦ Before submitting: pass every section knowledge check (100%).

Lesson Complete

You have successfully completed Data Cleaning and Descriptive Analyses. Your responses have been downloaded.

Regression takes the step from description to inference. The continuous outcomes summarised in this lesson can serve as dependent variables in regression, and the predictors cleaned and processed here as independent variables.