HSCI 410 · Lesson 7

Measurement and Psychometrics

Exploratory Data Analysis For Epidemiology

Learning objectives for this lesson:

  1. Describe the parts of a measure (construct, domain, item, instrument, scale, subscale and index), distinguish a reflective scale from a formative index, and explain why internal consistency statistics such as Cronbach's alpha apply only to a scale.
  2. Explain how item answers become scores: why reverse-worded items must be recoded according to the scoring manual before items are combined, how a missed reversal shows up in R output, and what a published cut-point gains and loses.
  3. Interpret reliability as the share of observed-score variance that is true-score variance, explain how random measurement error weakens observed associations, and identify which kind of reliability (test-retest, inter-rater or internal consistency) a study design can estimate, interpreting a given intraclass correlation or Cohen's kappa.
  4. Interpret Cronbach's alpha and McDonald's omega from R output, including corrected item-total correlations and alpha-if-item-deleted, judge whether a reliability is adequate for a stated use, and explain what alpha cannot show and what to do about a weak item.
  5. Explain validity as an argument for a stated use in a stated population, distinguish it from reliability, classify validity evidence (content, criterion, convergent, discriminant, known-groups and structural), and explain the minimal important difference and measurement invariance.
  6. Interpret an exploratory factor analysis (suitability, parallel analysis, rotation and loadings) and a confirmatory factor analysis (the chi-square test, the CFI, TLI, RMSEA and SRMR, standardized loadings and the comparison of one-factor and two-factor models), and explain what the results mean for how a scale is scored and reported.
  7. Explain the core ideas of item response theory (item difficulty, item discrimination, item characteristic curves and test information), and compare how item response theory and classical test theory describe precision.

This course was developed by Dr. Kiffer G. Card based on Dohoo, I. R., Martin, S. W., & Stryhn, H. (2012). Methods in Epidemiologic Research. VER Inc.

Lesson 7 · HSCI 410

Measurement and Psychometrics

How scales are built, how their reliability and validity are judged, and how latent variable models test their structure.

Exploratory Data Analysis For Epidemiology
  • You will prepare and score the De Jong Gierveld Loneliness Scale in R.
  • You will estimate its reliability and assemble evidence of its validity.
  • You will test whether its six items measure one construct or two.
Running case

Two questions from a reviewer

Question 1

Do the six De Jong Gierveld items measure one construct or two?

Question 2

Is the three-item emotional subscale reliable enough to analyze on its own?

Each section adds one part of the reply, and the last section drafts the revised Measures paragraph.

Lesson map

Four sections

1. From Constructs to Scores

This section covers vocabulary, scales and indices, scoring and classical test theory.

2. Reliability

This section covers kinds of reliability, Cronbach's alpha and its limits, and omega.

3. Validity

This section covers validity as an argument, the COSMIN taxonomy and construct evidence.

4. Latent Variable Models

This section covers exploratory and confirmatory factor analysis and item response theory.

R in this lesson

Four activities on the CSCS loneliness items

  • The activities load and score the scale, compute reliability and validity statistics, and fit factor models.
  • The packages are psych, GPArotation and lavaan.
  • The Factor Analysis and Scale Scoring walkthrough demonstrates the same steps line by line.
  • An optional box fits a graded response model with mirt, and it is not assessed.
How to work through the lesson

Watch, read, run, check

  • Each section has a narrated deck, reading, an R activity and a knowledge check.
  • Each knowledge check must be passed at 100% before the next section opens.
  • The final assessment has 16 questions.
  • The glossary and the deeper dive are available throughout the lesson.
Reference

Glossary: Key Terms, People & Concepts

📚 Reference page, available throughout the lesson

This glossary collects the key concepts, methods and people in this lesson. It can be used as a reference while working through the material or as a review before assessments. Typing in the search box filters the entries.

Key Concepts & Ideas
Construct A construct is a theoretical idea that a study wants to measure, such as loneliness or health literacy. It cannot be observed directly, so it is measured through the answers to several items.
Latent Variable A latent variable is an unobserved variable whose values are inferred from several observed indicators. Factor analysis and item response theory model constructs as latent variables.
Domain A domain is a distinct part of a construct, such as the emotional and social domains of loneliness. A scale that covers several domains usually has one subscale per domain.
Item and Indicator An item is a single question with its response options, and the answer is an indicator of the construct. The De Jong Gierveld scale has six items answered yes, more or less, or no.
Instrument An instrument is the complete measurement tool, such as a questionnaire, with its instructions and scoring rules. The CSCS questionnaire contains many scales.
Scale and Subscale A scale is a set of items whose answers are combined into a score for one construct. A subscale is a scale for one domain within a larger scale.
Index An index combines components that together define a construct, such as an area deprivation index or a social isolation index. Its components need not correlate.
Reflective Measurement Reflective measurement is a model in which the latent construct causes the answers to the items, so the items are expected to correlate. Internal consistency and factor analysis assume this model.
Formative Measurement Formative measurement is a model in which the components define the construct, so they need not correlate. Internal consistency statistics do not apply to formative indices.
Levels of Measurement Stevens described four levels of measurement: nominal (labels), ordinal (ordered), interval (equal spacing) and ratio (equal spacing with a true zero). Each De Jong Gierveld item is ordinal.
Scoring Rule and Cut-point A scoring rule states how item answers become a score, and a cut-point is a score at or above which a person is classified as having a condition. Both should be taken from the scale's documentation.
Reverse-Worded Item A reverse-worded item is worded in the opposite direction to the others, so its codes must be reversed (maximum plus minimum minus the original code) before the items are combined.
Acquiescence Acquiescence is the tendency of some respondents to agree with statements regardless of their content. Mixing positively and negatively worded items reduces its effect on scores.
Validity Validity is the degree to which evidence and theory support a particular interpretation and use of scores in a particular population. It is argued from several kinds of evidence.
Content and Face Validity Content validity is the degree to which items cover the construct, usually judged by experts. Face validity is whether the items look relevant to respondents and is the weakest form of evidence.
Criterion Validity Criterion validity is agreement between scores and a gold standard measured at the same time (concurrent) or later (predictive).
Construct Validity Construct validity is the degree to which scores relate to other variables as theory predicts. It includes convergent evidence (strong correlations with related measures), discriminant evidence (weaker correlations with distinct constructs) and known-groups comparisons.
Known-Groups Validity Known-groups validity is evidence that scores differ between groups that theory says should differ, such as people with few and many close friends.
Structural Validity Structural validity is the degree to which scores reflect the dimensions of the construct. It is assessed with factor analysis or item response theory.
Measurement Invariance Measurement invariance holds when the items relate to the construct in the same way in different groups. Without it, a difference in mean scores between groups may reflect the items.
Methods & Statistical Concepts
Classical Test Theory Classical test theory models each observed score as a true score plus random error (X = T + E). Reliability is the share of observed-score variance that is true-score variance.
Reliability Reliability is the proportion of the variance in observed scores that is true-score variance. It is estimated from consistency across occasions, raters or items.
Attenuation Attenuation is the weakening of an observed correlation by measurement error. The observed correlation equals the true correlation times the square root of the product of the two reliabilities.
Test-Retest Reliability Test-retest reliability is the consistency of scores when the same people are measured twice over a short interval. It is usually summarized with an intraclass correlation coefficient.
Inter-Rater Reliability Inter-rater reliability is agreement between two or more raters assessing the same people. Cohen's kappa is used for categories and the intraclass correlation coefficient for scores.
Internal Consistency Internal consistency is the degree to which the items of a scale move together on one occasion. It is the only kind of reliability that a single cross-sectional survey can estimate.
Intraclass Correlation Coefficient (ICC) The intraclass correlation coefficient measures agreement between repeated measurements of the same people. Unlike a Pearson correlation, it is lowered by a systematic shift between occasions. It is a different application of the same kind of variance ratio as the intracluster correlation coefficient used for clustered data.
Cohen's Kappa Cohen's kappa measures agreement between two raters on categories, corrected for chance: (po − pe) / (1 − pe), where po is the observed and pe the chance agreement. It ranges from −1 to 1, with 0 for chance agreement, and it is low when one category is rare.
Cronbach's Alpha Cronbach's alpha is a measure of internal consistency based on the item variances and the variance of the total score. It rises with the number of items and their average correlation, and it assumes equal loadings. A high alpha does not by itself show that the items measure one construct, which is tested by factor analysis, and values from 0.70 to 0.95 are a common acceptable range.
Corrected Item-Total Correlation The corrected item-total correlation is the correlation between one item and the sum of the other items (r.drop in psych::alpha()). Low values identify weak items.
Alpha if Item Deleted Alpha if item deleted is the value alpha would take without each item. It shows which items strengthen or weaken internal consistency.
Tau-Equivalence Tau-equivalence is the assumption that every item measures the construct equally well (equal loadings). When it fails, alpha underestimates reliability.
McDonald's Omega McDonald's omega is a reliability coefficient computed from a factor model, which allows items to have different loadings. It equals alpha when the loadings are equal.
Factor Loading A factor loading describes how strongly an item depends on a factor. A standardized loading is the item's correlation with the factor, and its square (the communality) is the share of the item's variance the factor explains.
Exploratory Factor Analysis (EFA) Exploratory factor analysis lets every item load on every factor and uses the data to suggest how many factors there are and which items belong together. It is fitted with psych::fa().
KMO and Bartlett's Test The Kaiser-Meyer-Olkin measure of sampling adequacy (values above about 0.6 are adequate) and Bartlett's test of sphericity check that items correlate enough for factor analysis.
Parallel Analysis Parallel analysis chooses the number of factors by keeping those whose eigenvalues exceed the eigenvalues from random data of the same size. It is run with fa.parallel().
Rotation Rotation turns a factor solution so that each item loads strongly on few factors. Oblique rotations such as oblimin let factors correlate, and orthogonal rotations such as varimax keep them uncorrelated.
Confirmatory Factor Analysis (CFA) Confirmatory factor analysis tests a structure stated in advance, in which each item loads only on its assigned factor. It is fitted with lavaan::cfa().
Fit Indices (CFI, TLI, RMSEA, SRMR) Fit indices judge how closely a confirmatory model reproduces the observed correlations. Common guides for good fit are CFI and TLI of 0.95 or higher, RMSEA of 0.06 or lower and SRMR of 0.08 or lower.
Item Response Theory (IRT) Item response theory models how each item behaves along the latent trait, with a curve for each item. It gives precision that varies along the trait.
Item Characteristic Curve An item characteristic curve shows the probability of the keyed answer at each level of the latent trait (theta).
Item Difficulty and Discrimination Difficulty (b) is the trait level at which the probability of the keyed answer is 0.5. Discrimination (a) is the steepness of the curve, which shows how sharply the item separates people near that level.
Rasch, 2PL and Graded Response Models The Rasch model gives every item the same discrimination, the two-parameter logistic (2PL) model lets it vary, and the graded response model extends the 2PL to items with ordered categories.
Test Information Test information measures how precisely a set of items locates people at each trait level. The standard error of a person's estimated trait is 1 divided by the square root of the information.
COSMIN Taxonomy The COSMIN taxonomy groups the measurement properties of health instruments into reliability, validity and responsiveness, and lists interpretability as an important characteristic.
Minimal Important Difference (MID) The minimal important difference is the smallest change in scores that people would regard as important. Anchor-based estimates link score changes to an external judgement; distribution-based estimates use half a standard deviation or the standard error of measurement, SD × √(1 − reliability).
Latent Class Analysis Latent class analysis is a latent variable model for a categorical latent variable. It assumes that people belong to a small number of hidden classes, each with its own answer probabilities, and gives each person a probability of belonging to each class.
Key People
Charles Spearman (1863–1945) Spearman was an English psychologist who introduced factor analysis and the correction for attenuation in 1904.
Lee J. Cronbach (1916–2001) Cronbach was an American educational psychologist who introduced coefficient alpha in 1951 and, with Paul Meehl, set out the concept of construct validity in 1955.
Roderick P. McDonald McDonald was a psychometrician whose work on factor analysis and test theory, including the book Test Theory: A Unified Treatment (1999), established the omega coefficient of reliability.
Samuel Messick (1931–1998) Messick was an American psychologist who described validity as a single, integrated judgement about the interpretation and use of scores.
Michael T. Kane Kane developed the argument-based approach to validation, in which the intended interpretation is stated and evidence is gathered for each claim it depends on.
Robert S. Weiss Weiss was an American sociologist who distinguished emotional loneliness (the absence of a close attachment) from social loneliness (the absence of a wider network) in 1973.
Jenny de Jong Gierveld De Jong Gierveld was a Dutch sociologist who developed, with Theo van Tilburg, the De Jong Gierveld Loneliness Scale and its six-item short form.
Georg Rasch (1901–1980) Rasch was a Danish mathematician who developed the one-parameter item response model that bears his name.
Fumiko Samejima Samejima was a psychometrician who developed the graded response model for items with ordered response categories in 1969.
No matching entries. Try a different search term.
Section 1 of 4

From Constructs to Scores

⏱ Estimated time: 45 minutes
Lesson 7 · Section 1

From Constructs to Scores

A score is useful only when the path from the idea being measured to the number in the dataset is clear.

Running case

A reviewer asks about the loneliness scale

The authors add the six De Jong Gierveld items into a single score. Do the items measure one construct or two? The emotional subscale has three items. Is it reliable enough to analyze on its own?Reviewer 2, on Amira's manuscript

Each section of the lesson adds one part of the reply, and Section 4 drafts the revised Measures paragraph.

Vocabulary

From construct to instrument

TermMeaningIn the running case
Construct (latent variable)The idea being measuredLoneliness
DomainA distinct part of the constructEmotional and social loneliness
Item (indicator)One question and its answer"I often feel rejected"
Scale and subscaleItems combined into a scoreTotal score 0 to 6; subscales 0 to 3
InstrumentThe whole toolThe CSCS questionnaire
Scales and indices

Which way do the arrows run?

Scale (reflective)

The construct causes the answers, so the items are expected to correlate.

An example is the De Jong Gierveld Loneliness Scale.

Index (formative)

The components define the construct, so they need not correlate.

Examples are an area deprivation index and the Steptoe social isolation index.

Internal consistency applies to scales and has no meaning for indices.

Recoding

Text answers become numbers, and three items are reversed

Stacked bar chart of the six De Jong Gierveld items after recoding, showing the percentage of 3,415 participants giving each answer, with higher codes meaning lonelier answers.
E marks the emotional items and S the social items; the social items were reverse-coded (new code = 4 − old code).
Scoring rules

The published rule turns answers into a score

  • Each item scores 1 for "More or less" or the lonely answer.
  • The emotional and social subscales each run from 0 to 3.
  • The total score runs from 0 (not lonely) to 6 (most lonely).
  • Amira's scores match the supplied score for all 3,415 people.
Bar chart of De Jong Gierveld total scores from 0 to 6 in the 2021 CSCS wave; the counts rise toward the upper end of the scale.
Total scores for 3,415 CSCS participants.
Classical test theory

Observed score = true score + error

The classical test theory model
\[ \color{#0B7B6B}{X} = \color{#6D28D9}{T} + \color{#C2410C}{E} \qquad \text{reliability} = \frac{\color{#6D28D9}{\sigma^2_T}}{\color{#0B7B6B}{\sigma^2_X}} = \frac{\sigma^2_T}{\sigma^2_T + \sigma^2_E} \]
X observed score T true score E random error

Reliability runs from 0 (all error) to 1 (no error).

Attenuation

Measurement error weakens correlations

Spearman's attenuation formula
\[ r_{\text{observed}} = r_{\text{true}} \times \sqrt{\text{rel}_X \times \text{rel}_Y} \]
Three scatter plots of simulated data with a true correlation of 0.6, showing the observed correlation falling as the reliability of both measures falls from 1.0 to 0.7 to 0.4.
Simulated data: as reliability falls, the cloud spreads out and the fitted line flattens.
Carry forward

What to take into the next section

  • A score is linked to its construct through domains, items, an instrument and a scoring rule.
  • The items of a scale are expected to correlate, whereas the components of an index need not.
  • Reliability is the true-score share of observed variance, and low reliability weakens correlations.

Overview

Many of the variables in health research describe things that cannot be observed directly. Loneliness, social support, depressive symptoms and health literacy are measured by asking people several questions and combining their answers into a score, and the quality of every later analysis depends on that score. This lesson takes the analyst's view of building and checking such a score, using the six items of the De Jong Gierveld Loneliness Scale, three of which are reverse-worded: how scales are built, how their reliability and validity are judged, and how factor analysis and item response theory test whether the items behave as the scale's authors intended. This first section sets out the vocabulary of measurement, separates scales from indices, describes levels of measurement and scoring rules, prepares and scores the loneliness items used throughout the lesson, and introduces classical test theory and its account of measurement error.

Learning Objectives

  • 1.1 Define construct, latent variable, domain, item, indicator, instrument, scale, subscale and index, and give an example of each from the De Jong Gierveld Loneliness Scale.
  • 1.2 Distinguish reflective measurement (a scale) from formative measurement (an index), and explain why internal consistency applies only to a scale.
  • 2.1 Describe levels of measurement, published scoring rules and cut-points, and explain what a cut-point gains and loses.
  • 2.2 Explain why reverse-worded items are recoded according to the scoring manual, and recode, reverse-code and score the six De Jong Gierveld items in R.
  • 3.1 State the classical test theory model, interpret a reliability value as the true-score share of observed-score variance, and explain how measurement error weakens a correlation.

1.1 The Vocabulary of Measurement

The lesson follows one running case, a manuscript on loneliness whose reviewer has raised two questions about how loneliness was measured. The case card below describes it, and each section of the lesson adds one part of the reply.

📋 Running case: a reviewer questions the loneliness scale

Amira, a graduate student, has written a manuscript on loneliness using the 2021 wave of the Canadian Social Connection Survey (CSCS). She measured loneliness with the six-item De Jong Gierveld Loneliness Scale and added the six items into a single score. Her manuscript has returned with this comment from the second reviewer: "The authors add the six De Jong Gierveld items into a single score. Do the items measure one construct or two? The emotional subscale has three items. Is it reliable enough to analyze on its own?" Each section of this lesson adds one part of Amira's reply. Section 1 prepares and scores the items, Section 2 estimates their reliability, Section 3 assembles evidence of validity, and Section 4 tests the structure of the scale and drafts the revised Measures paragraph.

Answering either question requires precise terms for the parts of a measure, because a reviewer will notice when a manuscript calls an index a scale or treats a single item as a construct. The terms are used with the following meanings throughout the course (Streiner, Norman & Cairney, 2015; DeVellis, 2017). A construct is the theoretical idea a study sets out to measure, such as loneliness. By this we mean that loneliness is not something that can be seen or counted directly, the way age or blood pressure can; it is an idea about an inner experience that researchers have defined in words. Because it cannot be observed, a construct is represented in statistical models by a latent variable, a variable whose values are never recorded in the data and are instead inferred from the answers people give. A domain is a distinct part of the construct. For example, a person can feel lonely because they lack one close confidant or because they lack a wider circle of friends, and these two experiences are treated as two domains of loneliness. An item is a single question with its response options, and the answer a person gives is an indicator of the construct, in the sense that it points toward, but does not by itself equal, the person's level of loneliness. The instrument is the complete questionnaire with its instructions and scoring rules. A scale combines the answers to several items into a score for one construct, a subscale does the same for one domain, and an index combines components that together define a construct.

Figure 7.1 shows how these terms connect for the De Jong Gierveld Loneliness Scale, from the construct at the top, through its two domains and six items, to the instrument and scoring rule at the bottom.

Construct Loneliness (latent) Domain Emotional loneliness Domain Social loneliness Emptiness Miss people Rejected Rely on Trust Feel close Items (indicators): three for each domain Instrument and scoring rule Six items in the CSCS questionnaire, scored 0 or 1 each: subscales 0 to 3, total 0 to 6
Figure 7.1. The chain from construct to score for the De Jong Gierveld Loneliness Scale. The two domains follow Weiss's (1973) distinction between emotional and social loneliness.

Each link in the chain has a fuller definition and an example from the CSCS. Click each of the six cards below to see how the term is used, for example which two domains Weiss (1973) described for loneliness and which other scales the CSCS instrument contains.

Construct and Latent VariableClick to explore
DomainClick to explore
Item and IndicatorClick to explore
InstrumentClick to explore
Scale and SubscaleClick to explore
IndexClick to explore

For any new questionnaire, the same chain can be traced from its description: what is being measured, what its parts are, which questions belong to which part, and how the answers are combined. Two of the terms, scale and index, describe measures that both combine several items into one number, and the next subsection explains why the difference between them decides which statistics can be used to judge a measure.

1.2 Scales and Indices: Reflective and Formative Measurement

A scale and an index rest on different models of how the items relate to the construct (Bollen & Lennox, 1991; Diamantopoulos & Winklhofer, 2001). In reflective measurement, the construct causes the answers. By this we mean that the person's level of loneliness comes first, and their answers to the items are a consequence of it. For example, a person who is lonely is more likely to report a sense of emptiness and more likely to report feeling rejected, because both answers flow from the same underlying experience. As a result, the answers to the items rise and fall together across people, and any one item could be replaced by a similar item without changing what is measured, since every item is simply another symptom of the same construct. In formative measurement, the direction is reversed: the components define the construct. A person is socially isolated because they are unmarried, rarely see family or friends and belong to no groups; isolation does not cause those circumstances, it is made up of them. This changes what we should expect from the data. Because nothing common causes the components, they can be unrelated to each other, and because each component is part of the definition, removing one changes the meaning of the index. A quick test is to ask whether having one component makes another more likely; if it does not, the measure is an index.

Figure 7.2 draws the two models side by side: the arrows run from the construct to the items in a reflective scale, and from the components to the construct in a formative index.

Reflective scale Loneliness Emptiness Rejected Miss people Items are expected to correlate. Formative index Unmarried Few friends No groups Social isolation Components need not correlate.
Figure 7.2. In a reflective scale the arrows run from the construct to the items. In a formative index they run from the components to the construct.

The three tabs below apply the distinction to three measures used in Canadian health research. Open each tab to compare the De Jong Gierveld scale, which is reflective, with an area deprivation index and the social isolation index in the CSCS file, which are formative.

The De Jong Gierveld Loneliness Scale (De Jong Gierveld & Van Tilburg, 2006) is reflective. Each item is a symptom of loneliness, so people who are lonelier are expected to give lonelier answers to every item. The items are interchangeable within a domain: "There are many people I can trust completely" and "There are enough people I feel close to" are two ways of asking about the same experience. Internal consistency, factor analysis and item response theory all assume this kind of model.

An area deprivation index, such as the Pampalon deprivation index used in Canadian health planning (Pampalon & Raymond, 2000), is formative. It combines census measures of education, employment and income with measures of household structure such as the share of people living alone. A neighbourhood is deprived because of these conditions, and the components need not move together: an area can have high employment and many people living alone. Each component is chosen because it is part of the definition, so dropping one changes what the index means.

The CSCS file contains the social isolation index of Steptoe and colleagues (2013), which gives one point each for being unmarried and not living with a partner, having less than monthly contact with children, with other family and with friends, and taking part in no groups or organizations. It is formative: a person who has no children is not thereby likely to belong to no clubs. Activity 7.2 in Section 2 shows that these components barely correlate, which is expected for an index and is no evidence against it.

The direction of the arrows matters because it decides how the quality of a measure is judged. The callout below states the rule, which returns in Section 2 when reliability is estimated.

Why the Distinction Matters for Analysis

Internal consistency statistics such as Cronbach's alpha (Section 2) measure how strongly the items of a scale move together. For a reflective scale, high correlations among items are expected, and low correlations suggest a problem. For a formative index, low correlations among components are normal, so an alpha computed for an index says nothing about its quality. The validity of an index is judged by whether its components cover the definition of the construct and whether the index relates to other variables as theory predicts.

A reflective scale and a formative index are therefore judged by different evidence. Either kind of measure must first be scored, and the next subsection turns to the levels of measurement and the scoring rules on which scoring depends.

2.1 Levels of Measurement, Scoring Rules and Cut-points

Before the answers to several items are combined, the analyst needs to know what kind of values the answers are and how the authors of the scale intended them to be scored. The first question concerns the level of measurement, and the second the scale's published scoring rule and any cut-point.

2.1.1 Levels of Measurement

Stevens (1946) described four levels of measurement. Nominal values are labels with no order, such as province of residence. Ordinal values have an order with unknown spacing, such as the answers "No", "More or less" and "Yes". Interval values have equal spacing without a true zero, such as temperature in degrees Celsius, and ratio values have equal spacing and a true zero, such as age or income. The level decides which summaries make sense: a mode for nominal values, a median for ordinal values, and means and differences for interval and ratio values.

Each De Jong Gierveld item is ordinal. A single Likert-type item is treated as ordinal because its few answers are ordered labels, and nothing shows that the steps from one answer to the next are equal. A sum of several ordinal items, however, is usually treated as approximately interval. By this we mean that analysts are willing to compute means of the total score and use it in linear regression, even though no single item has equal spacing. There are two reasons this works reasonably well. First, adding items spreads the scores over many possible values, so the total behaves more like a continuous measurement than any one item does. Second, the uneven spacing of each item's answers tends to average out across items: if one item has a large gap between two answers and another has a small gap, the total is not dominated by either. Because the justification depends on having many items and many possible values, the convention works well for long scales and is weaker when a score has only a few possible values, as the subscales here do (0 to 3), because a score with four possible values carries much of the uneven spacing of its items.

2.1.2 Scoring Rules and Cut-points

A published scale comes with a scoring rule, and the rule should be followed exactly so that scores can be compared with other studies. The De Jong Gierveld rule is dichotomous: each item scores 1 if the answer is "More or less" or the lonely answer, and 0 otherwise (De Jong Gierveld & Van Tilburg, 2006). The three emotional items are added to give the emotional subscale (0 to 3), the three social items give the social subscale (0 to 3), and the two subscales are added to give the total (0 to 6).

Some scales also publish a cut-point, a score at or above which a person is classified as having the condition. The CSCS file supplies a yes or no version of the total in which scores of 2 to 6 are classified as lonely. A cut-point gains a category that is easy to communicate, answers a question about how many people may need help and can be compared with other studies that use the same rule. It loses information, because it converts a score into a category: a person who scores 2 is treated the same as a person who scores 6, and scores of 1 and 2, which barely differ, fall in different groups. A cut-point should therefore be taken from the scale's documentation, stated in the manuscript, and used only when the research question needs a category.

The published rule gives a point for an answer of "More or less" or for the lonely answer on each item, so the lonely answer must be identified for every item. For three of the six De Jong Gierveld items the lonely answer is "No", and the next subsection explains how these items are recoded before they are combined.

2.2 Reverse-Worded Items and Scoring the Scale in R

Many scales mix negatively and positively worded items so that a respondent who agrees with every statement, a habit called acquiescence, does not receive an extreme score. The De Jong Gierveld scale does this. For the three emotional items ("I experience a general sense of emptiness", "I miss having people around me" and "I often feel rejected"), "Yes" is the lonely answer. For the three social items ("There are plenty of people I can rely on when I have problems", "There are many people I can trust completely" and "There are enough people I feel close to"), "No" is the lonely answer. Before the items can be combined, the positively worded items must be reverse-coded, as the scoring manual specifies, so that a higher number means a lonelier answer on every item.

Equation 7.1 gives the general rule for reversing the codes of an item, whatever the range of those codes.

Reverse-coding an item scored from a minimum to a maximum
\[ \color{#0B7B6B}{\text{reversed code}} = (\color{#6D28D9}{\text{maximum}} + \color{#6D28D9}{\text{minimum}}) - \color{#C2410C}{\text{original code}} \]Eq 7.1
For items coded 1 to 3, the reversed code is 4 minus the original code, so 1 becomes 3, 2 stays 2 and 3 becomes 1. For items coded 1 to 5, it is 6 minus the original code.

Leaving a reverse-worded item unreversed is one of the most common errors in survey analysis. The item then counts in the wrong direction, the total score mixes loneliness with its opposite, and every statistic computed from it is distorted; Section 2 shows in R how much the reliability of the scale falls when this step is skipped. A cross-tabulation of the original and reversed codes is a quick check that the reversal worked: every 1 should become 3, every 2 should stay 2 and every 3 should become 1. A second check compares the finished total with the score supplied with the data, which tests every step at once.

Activity 7.1 puts these steps into practice and prepares the data used throughout the lesson. It loads the 2021 wave of the CSCS, converts the text answers of the six items to numbers, applies Equation 7.1 to the three social items and checks the reversal with a cross-tabulation, then applies the published scoring rule, draws the bar chart of total scores and checks the total against the supplied score. Run the script alongside the activity and answer its questions from your own output.

R Activity 7.1: prepare and score the six loneliness items

Text version of the activity: the code of the run-along script, its console output and notes on reading it.

github_url <- "https://raw.githubusercontent.com/jorgeandr3s/heal/main/cscs/public_data/CSCS2025_full_cleaned_deidentified_data_and_metadata.RData"
load(url(github_url))   # survey data, read from the web: a data frame called data
dim(data)               # rows (survey responses) and columns
data <- data[data$SURVEY_collection_year == 2021, ]   # one row per person
dim(data)
Console output
[1] 13219 3247 [1] 4045 3247

The full file has 13,219 rows from several survey years. Keeping the 2021 wave leaves 4,045 people, one row per person, with the same 3,247 columns.

dj <- data.frame(
  emptiness = data$LONELY_dejong_emotional_social_loneliness_scale_emptiness,
  miss      = data$LONELY_dejong_emotional_social_loneliness_scale_miss,
  rejected  = data$LONELY_dejong_emotional_social_loneliness_scale_rejected,
  rely      = data$LONELY_dejong_emotional_social_loneliness_scale_rely,
  trust     = data$LONELY_dejong_emotional_social_loneliness_scale_trust,
  close     = data$LONELY_dejong_emotional_social_loneliness_scale_close)
table(dj$emptiness, useNA = "ifany")   # the answers are stored as text
table(dj$rely, useNA = "ifany")
Console output
Yes More or less No 1036 1484 962 Presented but no response <NA> 27 536 No Yes More or less 546 1409 1526 Presented but no response <NA> 28 536

The answers are stored as text. Besides "Yes", "More or less" and "No", 27 people saw the emptiness item without answering it ("Presented but no response") and 536 were not shown it (NA). The order in which the categories print comes from the file and has no meaning.

lv <- c("No", "More or less", "Yes")        # match() gives each answer's place
dj$emptiness_n <- match(dj$emptiness, lv)   # any other answer becomes NA
dj$miss_n      <- match(dj$miss, lv)
dj$rejected_n  <- match(dj$rejected, lv)
dj$rely_n      <- match(dj$rely, lv)
dj$trust_n     <- match(dj$trust, lv)
dj$close_n     <- match(dj$close, lv)
table(dj$emptiness, dj$emptiness_n, useNA = "ifany")   # check
Console output
1 2 3 <NA> Yes 0 0 1036 0 More or less 0 1484 0 0 No 962 0 0 0 Presented but no response 0 0 0 27 <NA> 0 0 0 536

match() gives the place of each answer in lv, so "No" becomes 1, "More or less" 2 and "Yes" 3; any other answer, such as "Presented but no response", becomes NA. The cross-tabulation is the check: each answer maps to exactly one number, and both kinds of missing answer are NA.

dj$rely_r  <- 4 - dj$rely_n
dj$trust_r <- 4 - dj$trust_n
dj$close_r <- 4 - dj$close_n
table(dj$rely_n, dj$rely_r)   # check: 1 -> 3, 2 -> 2, 3 -> 1
Console output
1 2 3 1 0 0 546 2 0 1526 0 3 1409 0 0

For the three social items "No" (code 1) is the lonely answer, so each is reversed with (maximum + minimum) − code = 4 − code. The table confirms the reversal for the rely item: the 546 people coded 1 now have 3, the 1,526 coded 2 stay at 2, and the 1,409 coded 3 now have 1, with no other cells filled.

items <- data.frame(emptiness = dj$emptiness_n, miss = dj$miss_n,
                    rejected = dj$rejected_n, rely = dj$rely_r,
                    trust = dj$trust_r, close = dj$close_r)
items <- na.omit(items)   # people who answered all six items
nrow(items)
Console output
[1] 3415

The items data frame holds the 3,415 people who answered all six items, with a higher code meaning a lonelier answer on every item. People coded "Presented but no response" were given NA by the recoding, so na.omit() removed them, together with the people who were not shown the items. Sections 2 and 4 analyze this data frame.

dj$emptiness_01 <- as.numeric(dj$emptiness_n >= 2)
dj$miss_01      <- as.numeric(dj$miss_n >= 2)
dj$rejected_01  <- as.numeric(dj$rejected_n >= 2)
dj$rely_01      <- as.numeric(dj$rely_r >= 2)
dj$trust_01     <- as.numeric(dj$trust_r >= 2)
dj$close_01     <- as.numeric(dj$close_r >= 2)

A comparison such as dj$emptiness_n >= 2 gives TRUE or FALSE, and as.numeric() turns TRUE into 1 and FALSE into 0. For "There are enough people I feel close to", the reversed code is 3 for "No" and 2 for "More or less", so dj$close_r >= 2 gives those two answers a point and "Yes" none.

dj$emotional_score <- dj$emptiness_01 + dj$miss_01 + dj$rejected_01
dj$social_score    <- dj$rely_01 + dj$trust_01 + dj$close_01
dj$total_score     <- dj$emotional_score + dj$social_score
table(dj$total_score, useNA = "ifany")
barplot(table(dj$total_score), col = "grey80",
        xlab = "Total score (0 = not lonely, 6 = most lonely)",
        ylab = "Number of participants")
Console output
0 1 2 3 4 5 6 <NA> 61 274 346 510 665 873 686 630
Bar chart of De Jong Gierveld total scores from 0 to 6 for 3,415 CSCS participants. The bars rise from 61 people at 0 and 274 at 1 to 873 at 5, with 686 at 6.

The table shows the total score for the 3,415 people with complete answers; the 630 people with any missing item have no total. Scores pile up toward the lonely end: 873 people score 5 and 686 score 6. Each item is ordinal, so adding the item scores treats the sum as approximately interval, a convention that is weaker for a subscale that can take only four values (0 to 3).

supplied <- data$LONELY_dejong_emotional_social_loneliness_scale_score
table(dj$total_score == supplied, useNA = "ifany")
Console output
TRUE <NA> 3415 630

All 3,415 totals equal the score supplied with the data, and the 630 NA values are the same people in both versions. This one comparison checks the recoding, the reversal and the scoring rule at once.

Files for this activity Data Code Answer key

R Reflect on what you just ran

Answer each question from your own output, as on the final exam: find the number, say what it means, judge the check.

1. Why does the De Jong Gierveld scale mix negatively and positively worded items, and why must rely, trust and close be reverse-coded before the six items are added? What does the table from table(dj$rely_n, dj$rely_r) show?

Model answerMixing negatively and positively worded items reduces the effect of acquiescence, the tendency of some respondents to agree with statements regardless of their content. For rely, trust and close, "No" (code 1) is the lonely answer, whereas "Yes" is the lonely answer on the emotional items; left unreversed, these three items would count in the wrong direction, the total would mix loneliness with its opposite, and every statistic computed from it would be distorted. The scoring manual therefore has them reverse-coded with (maximum + minimum) − original code, which for codes 1 to 3 is 4 − the code, so that a higher code means a lonelier answer on every item. The table shows that every original code maps to exactly one reversed code: the 546 people coded 1 now have 3, the 1,526 coded 2 stay at 2 and the 1,409 coded 3 now have 1, with no other cells filled, so the reversal worked.

2. Run your own code: the CSCS file classifies total scores of 2 to 6 as lonely. At the end of your script, type and run table(dj$total_score >= 2) and round(100 * prop.table(table(dj$total_score >= 2)), 1), which print how many of the 3,415 people have a total of 2 or more (TRUE) and what percentage that is. Report the result in one sentence, and give one advantage and one cost of reporting this category rather than the score.

Model answerR prints 3,080 for TRUE and 335 for FALSE, and 90.2% and 9.8%: 3,080 of the 3,415 people (90.2%) have totals of 2 to 6 and are classified as lonely. Advantage: a category is easy to communicate, answers a question about how many people may need help, and can be compared with other studies that use the same rule. Cost: it discards information, treating a score of 2 the same as a score of 6 and placing scores of 1 and 2, which barely differ, in different groups. The cut-point should come from the scale's documentation, be stated in the manuscript and be used only when the research question needs a category.

3. Run your own code: at the end of your script, type skip <- dj$emotional_score + (dj$rely_n >= 2) + (dj$trust_n >= 2) + (dj$close_n >= 2) and run it; it scores the three social items as they were stored, without the reversal. Then type table(skip == supplied, useNA = "ifany") and run it. A colleague says that once the scoring code runs without an error, there is no need to check the total against the score supplied with the data. Use this table and the one from line 66 to respond.

Model answerThe colleague is wrong: scored without the reversal, the code still runs without an error, yet only 676 of the 3,415 totals equal the supplied score and 2,739 differ. Comparing the total with the supplied score checks every step at once (the recoding of text to numbers, the reversal of the three social items and the scoring rule), so a single mistake shows up as totals that differ. With the reversal, all 3,415 totals match and the 630 missing values belong to the same people, so the score can be used with confidence and described in the Measures paragraph as scored by the published rule. If the check had failed, the fix would be to correct the coding as the scoring manual says, not to drop or change items.
Saved.

Figure 7.3 reproduces the bar chart of total scores that Activity 7.1 draws.

Bar chart of De Jong Gierveld total scores from 0 to 6. The bars rise from 61 people at 0 to 873 people at 5, with 686 at 6.
Figure 7.3. The bar chart of total scores drawn in Activity 7.1. The distribution of total scores is skewed toward the lonely end of the scale.

The six items are now coded in the same direction and scored by the published rule, the totals match the score supplied with the data, and Figure 7.3 shows that the total scores are skewed toward the lonely end of the scale. Even a correctly prepared score contains some error, and the next subsection introduces the model that classical test theory uses to describe it.

3.1 Classical Test Theory and Measurement Error

Classical test theory describes what an observed score contains and defines reliability. It also predicts what random error does to the associations estimated with a score, which is why the reliability of each subscale matters for Amira's results.

3.1.1 The Classical Test Theory Model

Classical test theory, set out formally by Lord and Novick (1968), is the simplest model of what an observed score contains. It states that each person's observed score X is the sum of a true score T and a random error E. The true score is the average score the person would obtain over many hypothetical repetitions of the measurement under the same conditions. By this we mean that if Amira could ask the same person the six loneliness items again and again, without the person remembering earlier answers or changing in the meantime, their scores would vary a little from one occasion to the next, and the average of all those scores would be their true score. The error is the difference between any one observed score and that average: a question misread, a passing mood, a guess between two answers. The model makes three assumptions about this error. Errors average zero, so they push some scores up and others down without a consistent direction. They are unrelated to the true score, so lonelier people are not more prone to error than less lonely people. And the errors of different measurements are unrelated to each other, so a person's bad day on the loneliness items does not also distort their answers on a depression scale. Because of these assumptions, error in this model is purely random, and the model says nothing about systematic error, which is a question of validity (Section 3).

Equation 7.2 states the model and the definition of reliability that follows from it.

The classical test theory model and the definition of reliability
\[ \color{#0B7B6B}{X} = \color{#6D28D9}{T} + \color{#C2410C}{E} \qquad\qquad \text{reliability} = \frac{\color{#6D28D9}{\sigma^2_T}}{\color{#0B7B6B}{\sigma^2_X}} = \frac{\sigma^2_T}{\sigma^2_T + \color{#C2410C}{\sigma^2_E}} \]Eq 7.2
An observed score equals a true score plus error. Reliability is the proportion of the variance in observed scores that is true-score variance. It runs from 0 (the scores are all error) to 1 (the scores contain no error).

A reliability of 0.80, for example, means that 80% of the variance in observed scores reflects real differences between people and 20% is random error. Notice that reliability is defined as a ratio of two variances, and a variance describes how much people differ from each other. Reliability is therefore a property of scores in a population, not a fixed property of the questionnaire, and it depends on how much people in that population differ. For example, suppose the same scale, with the same amount of random error in each answer, is given to two populations. In a population with a wide range of loneliness, the true-score variance is large, so the error makes up a small share of the total variance and reliability is high. In a population in which almost everyone is equally lonely, the true-score variance is small, so the same amount of error makes up a much larger share and reliability is lower. For this reason a manuscript reports the reliability observed in its own sample as well as the value from the scale's original publication.

3.1.2 Measurement Error Weakens Associations

Random measurement error has a predictable effect on correlations. Spearman (1904) showed that when two variables are measured with error, the correlation between the observed scores is smaller than the correlation between the true scores, a result called attenuation. Equation 7.3 gives Spearman's formula for the size of this effect.

Spearman's attenuation formula
\[ \color{#0B7B6B}{r_{\text{observed}}} = \color{#6D28D9}{r_{\text{true}}} \times \sqrt{\color{#C2410C}{\text{rel}_X} \times \color{#C2410C}{\text{rel}_Y}} \]Eq 7.3
The observed correlation equals the correlation between the true scores multiplied by the square root of the product of the two reliabilities. With a true correlation of 0.60 and reliabilities of 0.75 and 0.50, the expected observed correlation is 0.37, the value that Interactive 7.1 reports when its sliders are set to these numbers.

The reason for the weakening is that random error adds variation to each score that has nothing to do with the other variable. When a lonely person happens to score lower than their true score on one occasion, that error is not matched by a corresponding error in their depression score, so the points in a scatter plot spread out and the observed relationship looks looser than the true one. Notice also that the formula contains the two reliabilities but not the sample size. This means that a larger sample does not remove the weakening. A larger sample makes the observed correlation a more precise estimate, but it is a more precise estimate of the weakened value, because random error pulls the observed correlation toward zero however many people are measured. The formula can also be run in reverse. Dividing an observed correlation by √(relX × relY), for example with a line such as r_obs / sqrt(rel_x * rel_y) in R, estimates the correlation between the true scores, which is always at least as large as the observed one.

The corresponding result for a regression slope is simpler, but it depends on which variable contains the error. When the exposure is measured with random error, the observed slope equals the true slope multiplied by the reliability of the exposure, a factor called the regression dilution ratio. By this we mean that the slope is shrunk toward zero by the same proportion as the exposure's reliability: with an exposure reliability of 0.60, a true slope of 2.0 is observed as about 1.2. This happens because the error stretches the exposure along the horizontal axis without any matching change in the outcome, which flattens the fitted line. Random error in the outcome behaves differently. In a regression the outcome already has a residual term, and random error in the outcome is simply added to that residual. As a result, it leaves the slope unbiased, but because the residual variance is larger, it widens the confidence interval around the slope.

Interactive 7.1 lets you watch attenuation happen. It generates 250 people whose true loneliness and true depressive symptoms have a chosen correlation, then adds random error to each measure. Move the sliders to lower the reliability of either measure and watch the cloud of points spread and the observed correlation fall toward zero; the panel also reports the correlation expected from Equation 7.3, so that the simulated and expected values can be compared.

📊 Interactive 7.1: Simulation: watch a correlation weaken as measurement error grows

Observed correlation in these 250 people0.60
Expected from Spearman's formula0.60
Observed loneliness (X) Observed depression (Y)

The same weakening applies to the associations in Amira's manuscript, as the case card below explains.

📋 Running case: why the reviewer's question about reliability matters

Amira's manuscript reports a correlation between loneliness and depressive symptoms. If the emotional subscale has a reliability of about 0.5, as Section 2 will show, then Equation 7.3 implies that even a strong true association with a perfectly measured outcome would appear in her data at only about 0.71 of its true size, the square root of 0.5 (sqrt(0.5) in R). A weak or null finding for the emotional subscale could therefore reflect the measure as much as the world. This is the practical reason that the reliability of each subscale must be reported, and that a low value must be discussed when the subscale is analyzed on its own.

This section has defined the parts of a measure, separated reflective scales from formative indices, prepared and scored the six De Jong Gierveld items, and introduced the classical test theory model, in which reliability is the proportion of observed-score variance that is true-score variance and random error weakens associations. Because true scores are never observed, reliability has to be estimated from data, which is the task of Section 2. The key takeaways below draw the section together before its knowledge check.

Key Takeaways

  • 1.1 Every score you analyze sits at the end of a chain that starts with an idea. The construct is the idea a study sets out to measure, such as loneliness, and because it cannot be observed directly it is represented in models by a latent variable. A domain is a distinct part of the construct (emotional and social loneliness), an item is a single question with its response options, and the answer a person gives is an indicator of the construct; the instrument is the whole questionnaire with its instructions and scoring rules, and a scale (or a subscale, for one domain) combines items into a score. In the running case, the six De Jong Gierveld items form two three-item subscales that add up to a total from 0 to 6. When a question describes a new questionnaire, work down the chain and name each link from the question text: what is being measured, what its parts are, which questions belong to which part, and how the answers are combined. Keep the word "index" for a measure whose components define the construct rather than reflect it, which the next takeaway explains.
  • 1.2 Scales and indices both turn several items into one number, but they work in opposite directions, so the first question to ask is which way the arrows run. In a reflective scale the construct causes the answers: a lonelier person is more likely to report a sense of emptiness and more likely to feel rejected, so the items rise and fall together. In a formative index the components define the construct: a person counts as socially isolated because they are unmarried, rarely see friends and belong to no groups, and these circumstances need not be related to each other. The items of a reflective scale are expected to correlate, whereas the components of a formative index need not, so internal consistency applies only to scales. A quick test on an exam is to ask whether having one component makes another more likely (does caring for a grandchild make a person more likely to belong to a club?); if not, you are looking at an index, and low correlations among its components are expected rather than a sign of a problem.
  • 2.1 Before you combine answers, you need to know what kind of values they are and how the scale's authors meant them to be scored. A single item with a few ordered answers, such as "No", "More or less" and "Yes", is ordinal, because nothing shows that the steps between answers are equal; a sum of several items is usually treated as approximately interval, an assumption that is weaker when a score has only a few possible values, like the 0 to 3 subscales. Follow the published scoring rule exactly so that your scores can be compared with other studies: for the De Jong Gierveld scale, each item scores 1 for "More or less" or the lonely answer and 0 otherwise. A cut-point turns the score into a category (in the CSCS, totals of 2 to 6 count as lonely), which is easy to report and useful when the research question needs a category, but it discards information, because a person who scores 2 is treated the same as a person who scores 6. When a question asks what a cut-point gains and loses, give one of each in exactly that form, and add that the cut-point should be taken from the scale's documentation and stated in the manuscript.
  • 2.2 Many scales mix negatively and positively worded items so that a person who agrees with every statement (acquiescence) does not receive an extreme score, which means some items must be flipped before you add them up. Reverse-worded items must be recoded according to the scoring manual before items are combined. Decide which items to flip from the item wording and the manual, then apply one rule that works for any range of codes: the reversed code is (maximum + minimum) minus the original code, so on items coded 1 to 3 you subtract from 4 (as 4 - dj$rely_n does), and on items coded 1 to 5 you subtract from 6, so a 1 becomes 5 and a 4 becomes 2. If you skip this step, the unreversed items count in the wrong direction, the total mixes the construct with its opposite, and every statistic computed from the score is distorted. Always check the result with a cross-tabulation such as table(dj$rely_n, dj$rely_r): every 1 should become 3, every 2 should stay 2, every 3 should become 1, and no other cells should be filled.
  • 3.1 Classical test theory gives you a simple way to think about what any score contains: the observed score X is a true score T plus a random error E, where the error is everything that varies from one measurement to the next, such as a misread question or a passing mood. Reliability is the share of the variance in observed scores that is true-score variance, so it runs from 0 (all error) to 1 (no error), and a reliability of 0.80 means that 80% of the variance in observed scores reflects real differences between people and 20% is random error. Because it depends on how much people in a population differ, reliability belongs to scores in a particular sample, which is why a manuscript reports the value found in its own sample. Random error also weakens associations: when two measures are unreliable, the observed correlation is smaller than the correlation between the true scores (attenuation), and Spearman's formula gives the observed correlation as the true correlation times √(relX × relY). So if a question asks whether the true correlation is likely to be higher or lower than the one observed with imperfect scores, the answer is higher, and a weak or null finding with an unreliable measure may reflect the measure as much as the world.
Knowledge check: this section

1. A research team describes its new Caregiver Strain Questionnaire as follows: "Caregiver strain has two parts, physical strain and emotional strain. Ten statements, five for each part, are answered from 0 (never) to 3 (often). The answers to each set of five statements are added to give two scores, and the two scores are added to give a total." Which sentence uses the vocabulary of measurement correctly?

Adding two scores into a total does not make a measure an index. An index is defined by components that make up the construct, whereas here a person's caregiver strain is expected to drive the answers to all ten statements, which makes it a reflective scale.
Each term is matched to its role: the construct is the idea being measured, the domains are its distinct parts, an item is one statement with its answer options, and the statements combined for one domain form a subscale.
This reverses the levels. A single statement with its answer options is an item, not a subscale, and the questionnaire with its scoring rule is the instrument; the construct is the idea of caregiver strain that the instrument is built to measure.
Physical and emotional strain are domains, distinct parts of one construct, not separate instruments. The instrument is the whole questionnaire with its scoring rule, and a total score is a combination of subscales, not a domain.
The correct answer is b. The information needed is the role of each part of the description: caregiver strain is the idea the team wants to measure (the construct), physical and emotional strain are distinct parts of it (domains), each statement with its answer options is an item, the five statements for one domain are combined into a subscale, and the questionnaire with its scoring rule is the instrument. Option a is the tempting error: adding two subscale scores into a total does not make a measure an index. What makes a measure an index is that its components define the construct, whereas here a person's strain is expected to drive the answers to all ten statements. (Section 1, 1.1 The Vocabulary of Measurement, Figure 7.1 and the flip cards; Objective 1.1)

2. A researcher builds a household food insecurity measure by adding one point each for low household income, having no vehicle and living more than 2 km from a grocery store. The three components correlate only 0.05 to 0.15 with one another, and a colleague says that this shows the measure is poorly built. Which response is best?

Income, vehicle access and distance to a store are circumstances that make up food insecurity, so the arrows run from the components to the construct. Formative components need not move together, so low correlations are expected and internal consistency is not the standard for judging the index.
This treats the measure as reflective, but food insecurity does not cause a household's income or distance to a store; the components define it. Adding more strongly correlated components would chase internal consistency, which does not apply to a formative index.
Both scales and indices add their parts into one number, so adding is not what separates them. The direction of the arrows does: these components define food insecurity rather than being caused by it, so the colleague's criticism does not apply.
The classification is right, but the added threshold is not. Because formative components define the construct and need not correlate, no minimum correlation between them is required; an index is judged by whether its components cover the definition of the construct.
The correct answer is a. Decide which way the arrows run: low income, no vehicle and distance to a store are circumstances that make up a household's food insecurity, so the components define the construct (formative measurement), rather than the construct causing them as it causes the answers to a reflective scale. A household with a low income may still own a car, so the components need not move together, and their low correlations say nothing about the quality of the index. Option c is tempting because scales and indices both add items into one number, but adding is not what separates them; the direction of the arrows is. (Section 1, 1.2 Scales and Indices: Reflective and Formative Measurement, Figure 7.2 and the callout "Why the Distinction Matters for Analysis"; Objective 1.2)

3. A five-item stress scale is answered from 1 (never) to 5 (very often). Items 2 and 4 are worded positively (for example, "I felt on top of things") so that people who agree with every statement (acquiescence) do not receive extreme scores. The scoring manual says to reverse-code items 2 and 4, add the five items (5 to 25), and classify a score of 18 or more as high stress. An analyst scores the scale in R and prints the row for participant 17. Which statement is correct?

> d$s2_r <- 6 - d$s2          # reverse-code item 2
> d$s4_r <- 6 - d$s4          # reverse-code item 4
> d$total <- d$s1 + d$s2_r + d$s3 + d$s4_r + d$s5
> d$high <- d$total >= 18     # published cut-point
> d[17, c("s1","s2","s3","s4","s5","s2_r","s4_r","total","high")]
   s1 s2 s3 s4 s5 s2_r s4_r total high
17  4  2  5  1  3    4    5    21 TRUE
Adding the answers as stored would count the two positively worded items in the wrong direction. The scoring manual requires items 2 and 4 to be reversed before summing, and the reversed total that R prints is the one compared with the cut-point.
Subtracting from the highest code alone does not reverse a 1 to 5 scale correctly, because it maps a 1 to 4 instead of 5. The reversal rule is the maximum plus the minimum minus the original code, which is what the R lines apply.
The printed row shows that both reversed items now point in the same direction as the others, and the total meets the cut-point of 18. The category is convenient to report, but it discards differences in degree within the high-stress group.
The classification holds, but the claim about information is reversed. A cut-point turns a score into a category that treats everyone at or above 18 alike, so it discards the differences in degree that the continuous score keeps.
The correct answer is c. The information needed is the range of the codes (1 to 5), which items are reversed (2 and 4) and the cut-point (18 or more). The manual's rule is (maximum + minimum) minus the original code, which for codes 1 to 5 is 6 minus the code, as the two 6 - lines do. The printed row confirms that the reversal worked (the answer of 2 to item 2 is now 4 and the answer of 1 to item 4 is now 5), and R prints a total of 21 with high = TRUE, so the participant is at or above the cut-point. Option a adds the answers as stored, so the two positively worded items would count in the wrong direction, and option b uses 5 minus the code, which turns a 1 into 4 instead of 5. A cut-point gives a category that is easy to report and to compare with other studies, but it discards the differences in degree within each category, which is why option d is wrong. (Section 1, 2.1 Levels of Measurement, Scoring Rules and Cut-points and 2.2 Reverse-Worded Items and Scoring the Scale in R, Equation 7.1 and Activity 7.1; Objectives 2.1 and 2.2)

4. A study reports that its depression score has a reliability of 0.75 in its own sample. A student is asked what this value means. Which interpretation is correct?

Reliability is not a share of items. Under classical test theory it is the share of observed-score variance that reflects true differences between people, so 0.75 describes variance in scores, not how many items measure the construct.
Whether a score sorts people correctly into depressed and not depressed is a question of validity, judged against a criterion. Reliability concerns only random error, the share of observed-score variance that is true-score variance.
Reliability depends on how much people in a population differ in the construct, so the same scale can have different reliabilities in different samples. That is why a manuscript reports the reliability observed in its own sample.
Under classical test theory an observed score is a true score plus random error, and reliability is the share of observed-score variance that is true-score variance. The remainder is variance due to random measurement error.
The correct answer is d. Under classical test theory an observed score is a true score plus random error (X = T + E), and reliability is the share of the variance in observed scores that is true-score variance, so 0.75 means 75% true-score variance and 1 − 0.75 = 25% error variance. Option b is the tempting error: whether a score classifies people correctly is a question of validity, which reliability does not settle. Option c is wrong because reliability depends on how much people in the population differ, which is why a manuscript reports the reliability observed in its own sample. (Section 1, 3.1.1 The Classical Test Theory Model, Equation 7.2; Objective 3.1)

5. In a survey, the observed correlation between a loneliness score and a sleep-quality score is 0.36. The loneliness score has a reliability of 0.90 and the sleep-quality score has a reliability of 0.40. An analyst rearranges Spearman's attenuation formula in R to estimate the correlation between people's true levels of loneliness and sleep quality. Which interpretation is correct?

> 0.36 / sqrt(0.90 * 0.40)
[1] 0.6
This reverses the direction of attenuation. Random measurement error pulls observed correlations toward zero, so with unreliable scores the true association is expected to be stronger than the observed one, not weaker.
Random error does not average out of a correlation. Spearman's attenuation formula contains only the two reliabilities, not the sample size, so a large sample estimates the weakened correlation more precisely but does not remove the weakening.
Spearman's formula shows that random error weakens observed correlations in proportion to the reliabilities. Dividing the observed value by that factor, as the R line does, estimates the correlation between true levels, which is higher, so the reported 0.36 understates the association.
The value R prints is the estimated correlation between the true levels of loneliness and sleep quality, not a reliability. The reliabilities are inputs to the formula, and the result is not a share of the correlation that is error.
The correct answer is c. Equation 7.3 links the two correlations: observed r = true r × √(relX × relY). The R line divides the observed correlation by √(0.90 × 0.40), which undoes that weakening, and it prints 0.6, so the estimated correlation between the true levels is 0.60, well above the 0.36 observed. Option a reverses the direction of attenuation: random error pulls observed correlations toward zero, not away from it. Option b is the tempting error, because Spearman's formula contains only the reliabilities, so a larger sample does not remove the weakening. The point for interpretation is that an association estimated with unreliable scores understates the true association. (Section 1, 3.1.2 Measurement Error Weakens Associations, Equation 7.3 and Interactive 7.1; Objective 3.1)
● Complete the quiz to continue.
Section 2 of 4

Reliability

⏱ Estimated time: 40 minutes
Lesson 7 · Section 2

Reliability

How consistently does a score measure whatever it measures, and how should that consistency be reported?

Kinds of reliability

Consistent across what?

KindConsistency acrossUsual statistic
Test-retestTwo occasions, same peopleIntraclass correlation
Inter-raterTwo raters, same peopleKappa or intraclass correlation
Internal consistencyItems of one scale, one occasionCronbach's alpha, McDonald's omega

A single cross-sectional survey such as the CSCS supports only internal consistency.

Cronbach's alpha

Alpha depends on the number of items and how strongly they correlate

Standardized alpha
\[ \alpha_{\text{std}} = \frac{\color{#0B7B6B}{k}\,\color{#6D28D9}{\bar r}}{1 + (\color{#0B7B6B}{k} - 1)\,\color{#6D28D9}{\bar r}} \]
k number of items r̄ average inter-item correlation
0.75
social subscale (3 items, average r = 0.50)
0.51
emotional subscale (3 items, average r = 0.25)
Reading the output

The emotional subscale has one weak item

Emotional itemCorrected item-total rAlpha if item dropped
I experience a general sense of emptiness0.430.21
I miss having people around me0.160.64
I often feel rejected0.400.28

All three items together give alpha = 0.51.

Limits of alpha

What alpha assumes and what it cannot show

It assumes

Alpha assumes that every item measures the construct equally well (tau-equivalence).

When loadings differ, alpha underestimates reliability.

It cannot show

A high alpha does not establish that the items measure one construct.

Values above about 0.95 suggest redundant items, and alpha has no meaning for an index.

McDonald's omega

Omega allows the items to differ in quality

Omega from a one-factor model
\[ \omega = \frac{(\sum \color{#0B7B6B}{\lambda_i})^2}{(\sum \color{#0B7B6B}{\lambda_i})^2 + \sum \color{#C2410C}{\theta_i}} \]
λi loading of item i θi unique variance of item i
SubscaleLoadingsAlphaOmega
Social0.73, 0.73, 0.670.750.75
Emotional0.83, 0.20, 0.580.510.57
Reverse coding

Forgetting to reverse three items lowers alpha

0.59
alpha for six items, social items reversed
0.42
alpha for six items, social items left unreversed
  • The scoring manual decides which items are reversed, and check.keys = TRUE should not replace it.
  • A six-item alpha below that of one subscale is a first sign of two constructs.
Running case

The reliability part of Amira's reply

  • The social subscale has alpha = 0.75 and omega = 0.75, which is adequate for group comparisons.
  • The emotional subscale has alpha = 0.51 and omega = 0.57, which is low.
  • The published subscale is kept, the weak item is described, and a sensitivity analysis uses the two stronger items.
  • Associations with emotional loneliness are interpreted with measurement error in mind.
Carry forward

What to take into the next section

  • Reliability is consistency across occasions, raters or items, and one survey supports only internal consistency.
  • Alpha assumes equal loadings and cannot show unidimensionality, so omega is reported with it.
  • Reverse-worded items are recoded according to the scoring manual before items are combined.

Overview

Section 1 defined reliability as the proportion of the variance in observed scores that is true-score variance. This definition cannot be applied directly, because true scores are never observed. Reliability is therefore estimated from the consistency of a measurement: across two occasions, across two raters, or across the items of a scale. This section first sets out two conditions that must hold before the consistency of a set of items is worth estimating: the measure must be a scale rather than an index, and its items must be coded as the scoring manual says. It then describes the three kinds of reliability and the design that each requires, and teaches the most widely reported reliability statistic, Cronbach's alpha, together with its assumptions, its limits and its main alternative, McDonald's omega. Together these answer the second question in the reviewer's comment: whether the emotional subscale of the De Jong Gierveld scale is reliable enough to analyze on its own.

Learning Objectives

  • 1.3 Explain why alpha should not be reported for a formative index.
  • 2.3 Recognize a missed reversal in psych::alpha() output, and explain why the scoring manual, not check.keys = TRUE, decides which items are reversed.
  • 3.2 Compare test-retest, inter-rater and internal consistency reliability, and state which can be estimated from a single cross-sectional survey.
  • 3.3 Interpret an intraclass correlation for test-retest reliability and Cohen's kappa for agreement between two raters, including why a shift between occasions lowers the intraclass correlation and a rare category lowers kappa.
  • 4.1 Read Cronbach's alpha, corrected item-total correlations (r.drop) and alpha-if-item-deleted from psych::alpha(), and judge whether a reliability is adequate for comparing groups or for decisions about individuals.
  • 4.2 Explain the assumptions and limits of alpha, including why a high alpha does not show that a scale is unidimensional.
  • 4.3 Explain how McDonald's omega relaxes the equal-loadings assumption of alpha, and decide what to do about a weak item.

1.3 Internal Consistency Applies Only to Scales

A single survey such as the CSCS supports one kind of reliability, internal consistency: the degree to which the items of a measure move together on one occasion (3.2 explains why). Every internal consistency statistic, including Cronbach's alpha (4.1) and McDonald's omega (4.3), assumes the reflective model of Section 1, in which one construct causes all of the answers, so that the items are expected to correlate. The statistics therefore make sense only for a scale.

The components of a formative index define the construct and are not expected to correlate, so a low internal consistency for an index is expected and says nothing about its quality. The CSCS file contains such an index, the social isolation index of Steptoe and colleagues (2013), which gives one point each for being unmarried and not living with a partner, having less than monthly contact with children, with other family and with friends, and taking part in no groups or organizations. Activity 7.2, at the end of this section, shows that its five components correlate only between −0.10 and 0.36 and give an alpha of 0.34 for the 3,430 people with all five components. That value is printed only to show why it should not be reported as the reliability of the index. Omega would be equally out of place, because it also rests on a model in which one factor causes every answer. An index is judged instead by whether its components cover the definition of the construct and whether it relates to other variables as theory predicts.

When a low alpha is reported for a measure built by adding yes or no components, the first step is therefore to decide whether the measure is an index, and if it is, to say that alpha does not apply rather than to call the measure unreliable. The second condition concerns how the items of a scale are coded, which the next subsection takes up.

2.3 Spotting a Missed Reversal: The Scoring Manual Decides

Section 1 reverse-coded the three positively worded De Jong Gierveld items, as the scoring manual requires. If that step is skipped, three items count in one direction and three in the other, and every reliability statistic computed from the items is distorted: Activity 7.2 shows the six-item alpha falling from 0.59, with the correct coding, to 0.42 when rely, trust and close are left as they were stored.

The alpha() function in the psych package leaves three signs of a missed reversal, and they are worth looking for together. First, it prints a note and a warning that some items "probably should be reversed". Second, its item statistics include the corrected item-total correlation, printed as r.drop, which is the correlation of each item with the sum of the other items. A negative r.drop means that the item moves in the opposite direction to the rest of the scale: people who score high on the other items tend to score low on it. When that item is worded in the opposite direction to the others, the usual explanation is that it has not been reverse-coded. Third, alpha is lower than it should be. The remedy is to check the scoring manual, reverse-code the item if the manual says so, and recompute alpha, rather than to drop the item.

The warning also suggests rerunning alpha() with the option check.keys = TRUE, which flips every flagged item automatically. This option is tempting, but it should not be used to decide which items to reverse. The reason is that the software cannot read the wording of the items, so it cannot tell which ones are worded in the opposite direction. Instead, it works only from the numbers. It finds the first principal component of whatever data it is given, which is roughly the single direction along which the items vary together most, and it flags any item that correlates negatively with that direction. A miscoded item will usually be flagged, but so can an item that is correctly coded and merely relates weakly or inconsistently to the others. Even in the correctly coded CSCS data, psych warns that miss ("I miss having people around me") is negatively correlated with the first principal component, and check.keys = TRUE would reverse an item that is correctly coded and simply weak. The scoring manual, not check.keys = TRUE, decides which items are reversed. A negative r.drop for an oppositely worded item points to a coding error, whereas a low positive r.drop points to a weak item, which 4.1 describes.

With the measure confirmed as a scale and its items coded as the manual says, the next question is which kind of reliability the study design allows Amira to estimate.

3.2 Three Kinds of Reliability

Every reliability statistic measures consistency, and the kinds of reliability differ in what the consistency is across. The choice is driven by the main source of error in the measurement: the occasion on which it is made, the person who makes it, or the particular items used. Test-retest reliability asks whether the same people get similar scores on two occasions, inter-rater reliability asks whether two observers who rate the same people agree, and internal consistency asks whether the items of one scale move together on a single occasion.

Table 7.1 sets out the three kinds of reliability, the question each one answers, the design it needs and the statistic usually reported. When a study is described, its design decides which row applies.

Table 7.1. Three kinds of reliability, with the design and the usual statistic for each.

Kind of reliabilityQuestion it answersDesign neededUsual statistic
Test-retestDo the same people get similar scores on two occasions?The measure is given twice, with an interval short enough that the construct has not changed (often one to four weeks).Intraclass correlation coefficient (ICC) for scores; kappa for categories
Inter-raterDo two observers give the same rating to the same person?Two or more raters assess the same people independently.Cohen's kappa for categories; ICC for scores
Internal consistencyDo the items of one scale move together on one occasion?One administration of a multi-item scale.Cronbach's alpha; McDonald's omega

The design column is the one to read first, because a study can estimate only the kinds of reliability that its design supports. The callout below applies the table to the CSCS.

What the CSCS Can and Cannot Show

The 2021 wave of the CSCS asked the loneliness items once, in a self-completed online questionnaire with no interviewer or rater. Amira can therefore estimate internal consistency in her own sample, but she cannot estimate test-retest or inter-rater reliability. Evidence on the stability of the De Jong Gierveld scale over time has to come from the published literature on the scale, and her Measures paragraph should cite it as such and make clear which statistics were computed in her sample.

Although the CSCS cannot estimate them, test-retest and inter-rater reliability are reported for many instruments, and the next subsection explains how their two usual statistics are read.

3.3 The Intraclass Correlation and Cohen's Kappa

Test-retest reliability for a continuous score is usually summarized with an intraclass correlation, and agreement between two raters on categories with Cohen's kappa. Both statistics are read from R output, and each has one feature that is easy to misread.

3.3.1 The Intraclass Correlation for Test-Retest Reliability

The intraclass correlation coefficient (ICC) measures agreement as well as correlation. A Pearson correlation between two occasions can be high even if everyone scored five points higher the second time, because it asks only whether people keep the same rank order, whereas the intraclass correlation for absolute agreement asks whether people receive the same score twice and so counts such a shift as disagreement. In practice the intraclass correlation is read from R output: the ICC() function in the psych package prints it with a 95% confidence interval. Equation 7.4, the intraclass correlation for absolute agreement, is given in Worked Example 7.1 to show what the value compares.

Worked Example 7.1 applies the intraclass correlation to hypothetical scores for six people measured on two occasions, first as observed and then after a uniform shift in the second set of scores. Watch how the Pearson correlation and the ICC respond differently to the shift.

Worked Example 7.1: A test-retest intraclass correlation

Six people complete a symptom scale scored 0 to 20 on two occasions two weeks apart (hypothetical data). The intraclass correlation for absolute agreement compares the variation between people with the variation that remains: the disagreement between occasions for the same person, and any systematic shift from one occasion to the other.

PersonABCDEF
Occasion 1479121417
Occasion 25610111516
Intraclass correlation for absolute agreement (two-way model, single measurement)
\[ \text{ICC}_{A,1} = \frac{MS_R - MS_E}{MS_R + (k-1)\,MS_E + \frac{k}{n}\,(MS_C - MS_E)} \]Eq 7.4
MSR is the mean square between people, MSC the mean square between occasions, MSE the residual mean square, k the number of occasions (2) and n the number of people (6). These mean squares come from a two-way analysis of variance of the scores.

Both occasions have a mean of 10.5, so there is no shift between them (MSC = 0), and the people differ far more from each other than each person's two scores differ. For these scores, the ICC() function in the psych package reports an intraclass correlation for absolute agreement (the ICC2 row) of 0.98, and cor() gives a Pearson correlation of 0.97. Now suppose that every person scores 4 points higher on the second occasion. The Pearson correlation is unchanged at 0.97, because it ignores a shift that is the same for everyone. The shift enters Equation 7.4 through MSC, which rises from 0 to 48.0, and ICC() now reports 0.71, because people no longer receive the same score twice.

Koo and Li (2016) suggest reading an ICC below 0.50 as poor reliability, 0.50 to 0.75 as moderate, 0.75 to 0.90 as good and above 0.90 as excellent, preferably judged from its 95% confidence interval. The first version of the table shows excellent test-retest reliability and the shifted version moderate reliability. This intraclass correlation coefficient is a different application of the same kind of variance ratio as the intracluster correlation coefficient used for clustered data: there the clusters are clinics or people measured repeatedly, and here the repeated measurements of each person are the subject of interest.

The worked example shows why an ICC is the statistic for test-retest agreement: the shift left the Pearson correlation untouched but moved the ICC from excellent to moderate. Test-retest reliability also assumes that the construct itself did not change between the occasions. If something happens between them that changes people's true scores, for example an event that raises everyone's worry, part of the shift is real change rather than error, and the ICC then understates the stability of the measure. The timing of the two occasions should therefore be checked whenever a mean change between them is reported.

3.3.2 Cohen's Kappa for Agreement Between Raters

Inter-rater reliability for categories, the second row of Table 7.1, is usually summarized with Cohen's kappa, which R gives with the cohen.kappa() function in the psych package. When two raters classify the same people into categories, the proportion of people on whom they agree (the observed agreement, po) overstates their reliability, because some agreement would occur by chance. Kappa corrects for this using the expected chance agreement, pe, which is calculated from each rater's own proportions in each category. Equation 7.5 gives the formula.

Cohen's kappa
\[ \kappa = \frac{p_o - p_e}{1 - p_e} \]Eq 7.5
Kappa ranges from −1 to 1. A value of 0 means agreement no better than chance, 1 means perfect agreement, and negative values mean less agreement than chance.

Suppose two interviewers each classify the same 100 participants as lonely or not lonely. Both say lonely for 20 and not lonely for 65, so they agree on 85% of participants. The first interviewer calls 30% of participants lonely and the second 25%, which gives a chance agreement of 0.60. By this we mean that two interviewers who assigned labels at random, but in those same proportions, would still agree on about 60% of participants simply because both say "not lonely" most of the time. Kappa asks how much of the remaining room for agreement above chance the interviewers actually achieved, and cohen.kappa() returns a kappa of 0.625. A rare category lowers kappa further. This is because, when almost everyone is placed in the common category, two raters will agree on most people even if they are guessing, so chance agreement is high and there is little room left above it. If each interviewer calls only 6% of participants lonely and they agree on 92 of 100, the chance agreement is already 0.887 and kappa is only 0.29, although agreement looks high. Kappa is therefore always lower than the raw percentage agreement, and much lower when one category is rare.

Internal consistency remains the only kind of reliability that Amira can estimate from the CSCS. The next subsection introduces the statistic most often used to estimate it, Cronbach's alpha.

4.1 Cronbach's Alpha and the Item Statistics

Cronbach's alpha is the most widely reported estimate of internal consistency. The recall callout below returns to the alpha analysis from Lesson 2 and poses a retrieval question; try to answer it before revealing the answer, which the standardized formula in this subsection makes exact.

Recall: Lesson 2, Section 3 (Building Scales)

The R activity in Lesson 2, Section 3 computed Cronbach's alpha for the seven depression items of the course dataset (alpha = 0.904), together with the alpha-if-item-deleted values and the corrected item-total correlations (r.drop, from 0.65 to 0.74). Every alpha-if-item-deleted value was below 0.904, so every item contributed. This section adds the formula behind alpha, its standardized form and McDonald's omega, and applies them to the De Jong Gierveld items.

Retrieval question. Why does alpha rise when items are added to a scale, even when the new items correlate with the others no more strongly than the existing items do?

AnswerAlpha depends on the number of items as well as on how strongly they correlate. Each added item that reflects the construct adds more shared variation to the total score than item-specific noise, so a larger share of the total's variance is common to the items. The standardized formula below makes this exact.

Cronbach (1951) introduced coefficient alpha as a lower bound to the reliability of a sum of items, and it remains the statistic that reviewers most often expect to see. By a lower bound we mean that, under the classical test theory assumptions, the true reliability of the total score is at least as high as alpha, so alpha errs on the side of caution. Alpha works by comparing the variances of the individual items with the variance of the total score. The idea behind this comparison is that the variance of a sum depends on whether its parts move together. When the items move together, people who score high on one item tend to score high on the others, so their high answers pile up in the total while low scorers' low answers pile up too. The totals are then spread much more widely than the item variances alone would produce, and alpha approaches 1. When the items are unrelated, a high answer on one item is as likely to be paired with a low answer as with a high one, so the answers partly cancel out in the total. The variance of the total is then close to the simple sum of the item variances, and alpha approaches 0.

Equation 7.6 gives alpha in its raw form, computed from the variances of the items and of the total score, and in its standardized form, computed from the average correlation between pairs of items.

Cronbach's alpha (raw and standardized)
\[ \alpha = \frac{\color{#0B7B6B}{k}}{\color{#0B7B6B}{k} - 1}\left(1 - \frac{\sum \color{#C2410C}{\sigma^2_i}}{\color{#6D28D9}{\sigma^2_X}}\right) \qquad\qquad \alpha_{\text{std}} = \frac{\color{#0B7B6B}{k}\,\bar r}{1 + (\color{#0B7B6B}{k} - 1)\,\bar r} \]Eq 7.6
Here k is the number of items, σ2i is the variance of item i, σ2X is the variance of the total score, and r̄ is the average correlation between pairs of items. For the three social items, alpha() in Activity 7.2 prints an average_r of 0.5 and a std.alpha of 0.75; for the three emotional items, an average_r of 0.25 gives a std.alpha of 0.50.

The standardized formula shows that alpha depends on two things: the number of items and the strength of their correlations. Three items with an average correlation of 0.25 give an alpha of 0.50, but the same formula gives 0.80 for twelve items with the same average correlation. Short subscales therefore tend to have lower alphas than long scales, even when their items are of equal quality, which is part of the reason the three-item emotional subscale has a low value.

Several conventions are used to judge alpha, and each ties the value to a use. Nunnally (1978) suggested that about 0.70 suffices in the early stages of research, that about 0.80 is adequate for basic research comparing groups, and that 0.90 or higher is needed when scores are used to make decisions about individuals, such as a clinical cut-point; the widely used minimum of 0.70 comes from the first of these. Very high values, above about 0.90 to 0.95, can indicate that some items are close paraphrases of each other, which adds length without adding information (Streiner, 2003). These conventions are guides for judgement, and a manuscript should report the value and its confidence interval and let readers judge it in context.

Three parts of the alpha() output carry most of the information. The first line gives raw_alpha, with its 95% confidence interval below it; the "Reliability if an item is dropped" table shows what alpha would become without each item; and the r.drop column of the item statistics gives the corrected item-total correlations introduced in 2.3. In Activity 7.2, the social subscale has an alpha of 0.75 (Feldt 95% confidence interval 0.74 to 0.77), and dropping any one of its items would lower alpha, so every item contributes. The emotional subscale has an alpha of 0.51.

The item statistics also identify a weak item. A low positive r.drop (for example below about 0.3), together with an alpha that rises when the item is dropped, marks an item that relates only loosely to the others. In the emotional subscale, miss has an r.drop of 0.16, against 0.43 and 0.40 for the other two items, and alpha would rise from 0.51 to 0.64 without it. The table of response frequencies at the bottom of the output can show why an item is weak. When most people give the same answer, for example when two-thirds answer "never", the item has a floor effect (a ceiling effect when most give the top answer), so it separates people only at one end of the trait and correlates weakly with the other items. Whether such an item should be dropped is taken up in 4.3.

Reading these values correctly also requires knowing what alpha assumes and what it cannot show, which the next subsection examines.

4.2 What Alpha Assumes and What It Cannot Show

Alpha is easy to compute and is often over-interpreted (Sijtsma, 2009; Dunn, Baguley & Brunsden, 2014). Three limits matter most. Because alpha rises with the number of items, a long scale built from two related clusters of items can reach 0.80 or more, so a high alpha cannot show that a scale is unidimensional; that is a question for factor analysis (Section 4). A higher alpha is not always better, because values above about 0.95 usually mean that items repeat each other. And a reliable score is not thereby valid, because reliability concerns random error, not whether the score measures the intended construct.

The accordion below sets out five common misconceptions about alpha, each with the evidence from the CSCS loneliness items; two of them restate the points of 1.3 and 2.3 in the form a reviewer might raise them. Expand each panel in turn. The first panel also contains a callout that examines a common claim about alpha and asks you to rewrite it.

"A high alpha shows that the scale measures one construct"

Alpha measures how strongly the items correlate on average, and a set of items can correlate on average even when they measure two or more constructs. Because alpha rises with the number of items, a long scale built from two distinct clusters can reach 0.80 or more. In the CSCS data the six De Jong Gierveld items have an alpha of 0.59, lower than the 0.75 of the three social items alone, because the emotional and social items correlate only weakly with each other. Whether a scale is unidimensional is a question for factor analysis (Section 4).

A Common Claim: What an Alpha of 0.904 Shows

The seven depression items of the course dataset have an alpha of 0.904, and a report of that analysis might state: "An α of 0.904 indicates that the seven depression items measure essentially the same construct." Alpha shows how strongly the items correlate on average, which a set of items measuring two constructs can also do. The evidence that the depression items measure one construct comes from a parallel analysis of those items, which kept one factor, and from a one-factor solution, in which every loading was 0.69 or higher.

Question. Rewrite the quoted sentence so that it claims only what alpha shows.

Answer"An alpha of 0.904 indicates high internal consistency: the seven depression items correlate strongly on average, so little of the variation in the total score is item-specific." The claim that the items measure one construct rests on the parallel analysis and the one-factor solution, and it is stated separately with that evidence.
"A higher alpha is always better"

Above about 0.95, a high alpha usually means that items repeat each other, for example "I feel lonely" and "I feel alone". Redundant items lengthen the questionnaire and burden respondents, and they can narrow the construct to whatever the repeated items share. A good scale covers the breadth of its construct, which keeps the item correlations moderate.

"Alpha is appropriate for any multi-item measure, including indices"

Alpha assumes a reflective model in which one construct causes all of the answers. The components of a formative index are not expected to correlate, so a low alpha for an index is expected and says nothing about its quality. Activity 7.2 shows that the components of the Steptoe social isolation index in the CSCS correlate between −0.10 and 0.36 and give an alpha of 0.34, a value that should not be reported as a measure of the index's reliability.

"A reliable measure is therefore valid"

Reliability concerns random error, and validity concerns whether the score measures the intended construct. A bathroom scale that always reads two kilograms too heavy is perfectly consistent and systematically wrong. A loneliness scale could be highly reliable and still mainly measure general distress. Reliability limits validity, because a score that is mostly error cannot correlate strongly with anything, but it does not establish it. Section 3 takes up validity.

"Reverse-worded items can be summed without recoding"

If positively and negatively worded items are summed as stored, half of the items count in the wrong direction. Activity 7.2 shows that the six-item alpha falls from 0.59 to 0.42 when the three social items are left unreversed. The total score would be distorted in the same way, and every association estimated with it would be biased toward zero.

In addition to these limits, alpha rests on an assumption that is easy to overlook: that every item measures the construct equally well, a condition called tau-equivalence. By this we mean that each item is assumed to be just as strongly tied to the construct as every other item, so that no item is a better or worse indicator than the rest. Formally, each item is assumed to have the same loading on the common factor, a term that Section 4 explains in full. Real scales rarely meet this assumption exactly. When some items are stronger indicators than others, alpha treats them all as if they were average, and as a result it underestimates the reliability of the total score. For example, the assumption is plausible for the social items, whose loadings are similar, so alpha should be close to the true reliability of the social subscale. It is clearly false for the emotional items, where one item is much weaker than the other two, so alpha is likely to understate the reliability of the emotional subscale.

When the items differ in quality, a reliability coefficient that allows each item its own loading gives a better estimate of the reliability of the total score. The next subsection introduces such a coefficient, McDonald's omega.

4.3 McDonald's Omega and What to Do About a Weak Item

McDonald's omega (McDonald, 1999) estimates the reliability of a sum score from a factor model, which allows each item to have its own loading. This is the key difference from alpha. Instead of assuming that every item is an equally good indicator, omega first estimates how strongly each item is tied to the construct and then uses those separate estimates to judge the total score. For a scale with a single factor, omega is the share of the variance of the total score that is explained by the factor. By this we mean that omega splits the differences between people's total scores into a part that comes from the construct the items share and a part that comes from each item's unique variance, and reports the first part as a proportion. This matches the classical definition of reliability in Section 1, with the factor playing the role of the true score. Equation 7.7 gives omega for a one-factor model.

McDonald's omega (total) for a one-factor model
\[ \omega = \frac{\left(\sum \color{#0B7B6B}{\lambda_i}\right)^2}{\left(\sum \color{#0B7B6B}{\lambda_i}\right)^2 + \sum \color{#C2410C}{\theta_i}} \]Eq 7.7
The loadings λi are the standardized loadings of the items on one factor, and the unique variances θi equal 1 minus each squared loading. Activity 7.2 applies the formula with the line sum(lambda)^2 / (sum(lambda)^2 + sum(f_soc$uniquenesses)), which prints 0.754 for the social items.

When the loadings are equal, omega equals alpha, and when they differ, omega is the better estimate. Activity 7.2 computes omega for both subscales from a one-factor model fitted with fa(), which Section 4 explains in detail. The social loadings are similar (0.73, 0.73 and 0.67), and alpha and omega are both 0.75. The emotional loadings are very unequal (0.83, 0.20 and 0.58), so omega (0.57) is higher than alpha (0.51). Both values are low. When omega is noticeably above alpha, one item usually has a much weaker loading than the others. Omega is now recommended alongside or in place of alpha in many fields, and reporting both lets readers compare a study with earlier work that reported only alpha.

The weak item still has to be dealt with. The case card below shows how Amira decides what to do about miss, and the same reasoning applies to a weak item in any published scale.

📋 Running case: deciding what to do about a weak item

The alpha output shows that the emotional item "I miss having people around me" has a corrected item-total correlation of 0.16, and that alpha for the emotional subscale would rise from 0.51 to 0.64 if it were dropped. It is tempting to delete the item and report the higher value. Amira decides against making that her main analysis, for three reasons. The published scoring rule uses all three items, so a two-item subscale would not be comparable with other studies of the scale. An alpha chosen after looking at the data overstates reliability in new samples. The item also covers a part of emotional loneliness (missing the presence of others) that the other two items do not. Her reply therefore keeps the published subscale as the main measure, reports alpha and omega for it, names the weak item, and adds a sensitivity analysis that repeats her main model with the two-item version. If the conclusions agree, the weak item has not driven them.

Keeping the published subscale as the main measure, reporting alpha and omega, naming the weak item and testing it with a sensitivity analysis protects both the comparability of the results and the honesty of the reliability estimate. The card below links to a narrated R walkthrough of the same reliability analysis, which you may find useful before running the activity.

Learn to do this in R

Narrated R walkthrough: Factor Analysis and Scale Scoring

The Factor Analysis and Scale Scoring walkthrough runs the same reliability analysis line by line with narration, from recoding the items to the effect of a missed reversal. It is a useful companion to Activity 7.2 for anyone who would like to see each step demonstrated before running it.

Open the Factor Analysis and Scale Scoring walkthrough

Activity 7.2 brings the section together in R. It computes alpha and the item statistics for both subscales, omega from one-factor models, the correlations and alpha of the Steptoe index, and the six-item alpha with and without the reversal. Run the script alongside the activity and answer its questions from your own output.

R Activity 7.2: Cronbach's alpha, omega and a missed reversal

Text version of the activity: the code of the run-along script, its console output and notes on reading it.

github_url <- "https://raw.githubusercontent.com/jorgeandr3s/heal/main/cscs/public_data/CSCS2025_full_cleaned_deidentified_data_and_metadata.RData"
load(url(github_url))                      # survey data, read from the web
data <- data[data$SURVEY_collection_year == 2021, ]   # the 2021 wave
dj <- data[, paste0("LONELY_dejong_emotional_social_loneliness_scale_",
                    c("emptiness", "miss", "rejected", "rely", "trust", "close"))]
names(dj) <- c("emptiness", "miss", "rejected", "rely", "trust", "close")
answers <- c("No", "More or less", "Yes")  # coded 1, 2 and 3; anything else NA
stored <- na.omit(data.frame(lapply(dj, match, table = answers)))
soc <- c("rely", "trust", "close")         # the positively worded items
items <- stored
items[, soc] <- 4 - stored[, soc]          # reverse: 1 becomes 3, 3 becomes 1
nrow(items)                                # people who answered all six items
Console output
[1] 3415

The file is read straight from the web, and the 2021 wave gives one row per person. match() turns each text answer into its position in answers (No = 1, More or less = 2, Yes = 3), so any other answer, such as "Presented but no response", becomes NA, and na.omit() keeps the 3,415 people who answered all six items. stored holds the codes as answered; items reverses the three positively worded social items, so a higher code means a lonelier answer on every item. Alpha and omega are therefore computed from the three-point item codes (1 to 3), whereas the published subscale and total scores use the 0 or 1 rule of Section 1.

library(psych)                             # alpha() and fa()
social_items    <- items[, soc]
emotional_items <- items[, c("emptiness", "miss", "rejected")]
alpha(social_items)
a_emo <- alpha(emotional_items)
round(a_emo$total, 2)                      # overall alpha only
round(a_emo$alpha.drop[, 1:2], 2)          # alpha if each item is dropped
round(a_emo$item.stats[, "r.drop", drop = FALSE], 2)   # corrected item-total r
Console output
Reliability analysis Call: alpha(x = social_items) raw_alpha std.alpha G6(smc) average_r S/N ase mean sd median_r 0.75 0.75 0.67 0.5 3.1 0.0073 1.8 0.59 0.49 95% confidence boundaries lower alpha upper Feldt 0.74 0.75 0.77 Duhachek 0.74 0.75 0.77 Reliability if an item is dropped: raw_alpha std.alpha G6(smc) average_r S/N alpha se var.r med.r rely 0.66 0.66 0.49 0.49 1.9 0.012 NA 0.49 trust 0.66 0.66 0.49 0.49 2.0 0.012 NA 0.49 close 0.69 0.70 0.53 0.53 2.3 0.010 NA 0.53 Item statistics n raw.r std.r r.cor r.drop mean sd rely 3415 0.82 0.83 0.69 0.60 1.8 0.71 trust 3415 0.83 0.82 0.68 0.59 1.8 0.74 close 3415 0.81 0.81 0.64 0.56 1.7 0.72 Non missing response frequency for each item 1 2 3 miss rely 0.40 0.44 0.16 0 trust 0.37 0.42 0.21 0 close 0.44 0.40 0.16 0 raw_alpha std.alpha G6(smc) average_r S/N ase mean sd median_r 0.51 0.5 0.44 0.25 1.01 0.01 2.06 0.52 0.16 raw_alpha std.alpha emptiness 0.21 0.21 miss 0.64 0.64 rejected 0.28 0.28 r.drop emptiness 0.43 miss 0.16 rejected 0.40

Reading the output. The first row gives raw_alpha 0.75 (computed from the item codes) and std.alpha 0.75 (computed from the correlations), with an average_r of 0.5. The Feldt 95% confidence interval runs from 0.74 to 0.77. Under Reliability if an item is dropped, removing any one social item lowers alpha to 0.66 to 0.69, so every item contributes. Under Item statistics, r.drop is the corrected item-total correlation, the correlation of each item with the sum of the other two: 0.60, 0.59 and 0.56. The last table shows how often each answer was given. The $total, $alpha.drop and $item.stats parts of the result can be printed separately: for the emotional subscale, raw alpha is 0.51 with an average inter-item correlation of 0.25, dropping miss would raise alpha to 0.64 whereas dropping either other item would lower it to about 0.2 to 0.3, and the corrected item-total correlation for miss is 0.16, against 0.43 and 0.40 for the other two items.

f_soc  <- fa(social_items, nfactors = 1)   # one-factor model
lambda <- f_soc$loadings[, 1]              # the three loadings
round(lambda, 2)
sum(lambda)^2 / (sum(lambda)^2 + sum(f_soc$uniquenesses))   # omega
f_emo  <- fa(emotional_items, nfactors = 1)
lambda <- f_emo$loadings[, 1]
round(lambda, 2)
sum(lambda)^2 / (sum(lambda)^2 + sum(f_emo$uniquenesses))
Console output
rely trust close 0.73 0.73 0.67 [1] 0.7541436 emptiness miss rejected 0.83 0.20 0.58 [1] 0.5682014

fa(..., nfactors = 1) fits a one-factor model and returns the loadings and the unique variances ($uniquenesses). The last line of each block applies the omega formula (Equation 7.7). For the social items, omega is 0.754, the same as alpha, because the three loadings are similar. For the emotional items, omega is 0.568, higher than the alpha of 0.51, because the loading of miss (0.20) is far below the others. The omega() function in psych gives the same values.

iso <- data[, paste0("LONELY_steptoe_isolation_index_",
                     c("unmarried", "clubs", "friends", "kids", "other_fam"), "_num")]
names(iso) <- c("unmarried", "clubs", "friends", "children", "family")
iso <- na.omit(iso)
round(cor(iso), 2)
round(alpha(iso)$total[, 1:2], 2)          # shown only to explain why it is not reported
Console output
unmarried clubs friends children family unmarried 1.00 -0.10 0.06 0.12 0.12 clubs -0.10 1.00 0.04 0.09 0.07 friends 0.06 0.04 1.00 0.15 0.36 children 0.12 0.09 0.15 1.00 0.20 family 0.12 0.07 0.36 0.20 1.00 raw_alpha std.alpha 0.34 0.38

The five components of the Steptoe social isolation index correlate only weakly (from −0.10 to 0.36), and alpha is 0.34 for the 3,430 people with all five components. For a formative index this is expected: being unmarried does not make a person more likely to have little contact with friends. The value is printed here only to show why it should not be reported as the reliability of an index.

round(alpha(items)$total[, 1:2], 2)
Console output
Some items ( miss ) were negatively correlated with the first principal component and probably should be reversed. To do this, run the function again with the 'check.keys=TRUE' option raw_alpha std.alpha 0.59 0.59 Warning message: Some items were negatively correlated with the first principal component and probably should be reversed. To do this, run the function again with the 'check.keys=TRUE' option

With the social items correctly reversed, alpha for all six items is 0.59, lower than the 0.75 of the social items alone. psych prints a note and then a warning that miss is negatively correlated with the first principal component: miss is correctly coded but relates weakly, and slightly negatively, to the social items. The suggested option check.keys = TRUE would flip flagged items automatically, which here would reverse a correctly coded item. Which items to reverse is decided by the scoring manual, and the warning is a prompt to check the coding. Question 3 asks you to run alpha(stored), the six items as answered, before the reversal.

Files for this activity Data Code Answer key

R Reflect on what you just ran

Answer each question from your own output, as on the final exam: find the number, say what it means, judge the check.

1. Report alpha for the social and the emotional subscale, with the 95% confidence interval where it is printed. Is the reliability of each subscale adequate for comparing groups of participants in research? Would either be adequate for deciding which individual participants are offered a loneliness support programme? Explain.

Model answerThe social subscale has alpha = 0.75 (Feldt 95% CI 0.74 to 0.77), and the emotional subscale has alpha = 0.51. For comparing groups of participants in research, the social subscale is adequate: 0.75 is above the 0.70 that suffices in early research, though just below the 0.80 adequate for basic research. The emotional subscale is below 0.70, so its reliability is low, and associations estimated with it will be weakened by random measurement error. For decisions about individuals, such as who is offered a support programme, neither subscale is adequate, because that needs 0.90 or higher.

2. Describe how the item miss ("I miss having people around me") behaves, using the alpha() output, and explain why omega is higher than alpha for the emotional subscale but equal to alpha for the social subscale. A colleague proposes reporting the two-item emotional subscale (alpha 0.64) as the main measure. What should be done instead, and why?

Model answerThe item miss is weak: its corrected item-total correlation (r.drop) is only 0.16, against 0.43 and 0.40 for the other two items, and alpha would rise from 0.51 to 0.64 without it, whereas dropping either other item would lower alpha (to 0.21 or 0.28). Alpha assumes that every item has the same loading (tau-equivalence); the social loadings are similar (0.73, 0.73 and 0.67), so alpha and omega are both 0.75, but the emotional loadings are very unequal (0.83, 0.20 and 0.58), so alpha (0.51) underestimates reliability and omega (0.57) is the better estimate. Keep the published three-item subscale as the main measure, report alpha and omega, name miss as a weak item, and run a sensitivity analysis with the two stronger items; if the conclusions agree, miss has not driven them. The published scoring rule uses all three items, so a two-item score would not be comparable with other studies, and an alpha chosen after looking at the data overstates reliability in new samples.

3. Run your own code: at the end of the script, type alpha(stored) and run it. The data frame stored holds the six items as they were answered, before rely, trust and close were reversed. Which items does psych flag, what do their r.drop and alpha-if-dropped values suggest, and why has this happened? A colleague suggests rerunning alpha() with check.keys = TRUE, or dropping the flagged items: what should be done instead, and why? Finally, after the correct reversal, alpha for the six items is 0.59: does this show that the six items measure a single construct?

Model answerAlpha falls to 0.42 (from 0.59 with the correct coding), and psych flags emptiness and rejected: their corrected item-total correlations (r.drop) are close to zero (0.046 and 0.064), and alpha would rise to 0.47 and 0.46 without them, so they appear to move against the rest of the items. This has happened because rely, trust and close are positively worded and have not been reverse-coded as the scoring manual requires, so three items count in one direction and three in the other; psych cannot tell which half is wrong, and it flags the items that run against the three strongly correlated social items. Reverse rely, trust and close as the scoring manual says and recompute alpha, rather than rerunning with check.keys = TRUE or dropping items: check.keys = TRUE would reverse emptiness and rejected (the wrong items), and in the correctly coded data it would reverse miss, a correctly coded item that is simply weak, while dropping items would remove good items to hide a coding error. No: alpha shows how strongly the items hang together on average and rises with the number of items, so it cannot show that the items measure a single construct; here the six-item alpha (0.59) is even lower than that of the three social items alone (0.75), and whether one or two constructs underlie the items is a question for factor analysis, since the scale was designed with two subscales.
Saved.

Your output from Activity 7.2 contains every value interpreted in this section, from the alpha of 0.34 for an index to the weak loading of miss and the effect of a missed reversal.

This section has shown that internal consistency applies only to scales, that a missed reversal is fixed from the scoring manual, that a single survey supports only internal consistency, that the social subscale of the De Jong Gierveld scale has adequate internal consistency (alpha and omega of 0.75), and that the emotional subscale has low internal consistency (alpha of 0.51 and omega of 0.57). It has also shown that alpha must be read with its assumptions in mind. A reliable score can still measure the wrong construct, so Section 3 turns to validity. The key takeaways below draw the section together before its knowledge check.

Key Takeaways

  • 1.3 Alpha assumes a reflective model, in which one construct causes all of the answers, so it only makes sense for a scale. The components of a formative index define the construct and are not expected to correlate, so a low alpha for an index is expected and says nothing about its quality: the five components of the Steptoe social isolation index correlate between −0.10 and 0.36 and give an alpha of 0.34, which is not reported as its reliability. When a question shows a low alpha for a measure built by adding yes or no components, such as being unmarried or belonging to no club, first decide whether it is an index, and if it is, say that alpha does not apply rather than calling the measure unreliable. An index is judged instead by whether its components cover the definition of the construct and whether it relates to other variables as theory predicts.
  • 2.3 A missed reversal leaves clear fingerprints in alpha() output, and learning to spot them protects you from one of the most common errors in survey analysis. Look for three signs together: psych's warning that some items "probably should be reversed", a negative r.drop for an item worded in the opposite direction to the others (people who score high on the rest of the scale score low on it), and an alpha lower than it should be (0.42 instead of 0.59 when the three social items were left unreversed). The fix is to check the scoring manual, reverse-code the item if the manual says so and recompute alpha, not to drop the item. Do not let check.keys = TRUE make the decision for you, because it flips whatever is flagged, and in Activity 7.2 it would have reversed miss, a correctly coded item that is simply weak. The scoring manual, not check.keys = TRUE, decides which items are reversed, and a low positive r.drop points to a weak item rather than a coding error.
  • 3.2 Reliability always means consistency, so the first question to ask is "consistent across what?" Test-retest reliability asks whether the same people get similar scores on two occasions a short interval apart, inter-rater reliability asks whether two observers who rate the same people agree, and internal consistency asks whether the items of one scale move together on a single occasion. Each kind needs its own design: two administrations of the measure, two or more raters, or one administration of a multi-item scale. When a question describes a study, read its design before you name a statistic. A single cross-sectional survey such as the CSCS, asked once and with no raters, supports only internal consistency, so evidence on stability over time has to come from the scale's published studies and should be cited as such.
  • 3.3 For test-retest reliability of a score, the intraclass correlation (ICC) asks whether people receive the same score twice, not just whether they keep the same rank order. That is why a shift in everyone's scores between occasions lowers the ICC even when the Pearson correlation is unchanged: in Worked Example 7.1, adding 4 points to every second score leaves the Pearson correlation at 0.97 but lowers the ICC from 0.98 to 0.71. Describe an ICC with Koo and Li's bands (below 0.50 poor, 0.50 to 0.75 moderate, 0.75 to 0.90 good, above 0.90 excellent), and if a question reports a mean change between occasions, say that the ICC counts the shift as disagreement and check the timing, because test-retest reliability assumes the construct itself did not change. For two raters classifying people into categories, Cohen's kappa removes the agreement expected by chance, so 0 means agreement no better than chance and 1 means perfect agreement. Kappa is therefore lower than the raw percentage agreement, and much lower when one category is rare, because chance agreement on the common category is then high: in the second kappa example of 3.3, 92% agreement gives a kappa of only 0.29 when each interviewer calls just 6% of people lonely.
  • 4.1 When you run alpha() from the psych package, three parts of the output carry most of the information. The first line gives raw_alpha, with its 95% confidence interval below it; the "Reliability if an item is dropped" table shows what alpha would become without each item; and the r.drop column of the item statistics is the corrected item-total correlation, the correlation of each item with the sum of the other items. Read them together: an item with a low r.drop (below about 0.3) whose removal would raise alpha is a weak item, as miss is in the emotional subscale (0.16, with alpha rising from 0.51 to 0.64 without it), and the response frequencies at the bottom of the output can show why, for example a floor effect when most people give the same lowest answer. To judge whether a value is adequate, always tie it to the use: about 0.70 suffices in the early stages of research, about 0.80 is adequate for basic research comparing groups, and 0.90 or higher is needed for decisions about individuals, such as classifying a person with a clinical cut-point. A strong exam answer gives the value, the guide and the use in one sentence, for example "an alpha of 0.79 is adequate for comparing groups but too low for decisions about individual members".
  • 4.2 Alpha is easy to compute and easy to over-read, so it helps to know exactly what it can and cannot tell you. Cronbach's alpha rises with the number of items, assumes equal loadings and cannot show unidimensionality, so McDonald's omega is reported alongside it. Because alpha depends on the number of items as well as on their average correlation, a long scale built from two related clusters of items can still reach 0.80 or more, so a high alpha never answers the question "one construct or two?"; that question belongs to factor analysis. Higher is not always better either, because values above about 0.95 usually mean that some items are near copies of each other. A common exam trap pairs a high alpha with a parallel analysis that suggests two factors: the two findings do not contradict each other, because alpha shows how strongly the items hang together on average, not how many things they measure.
  • 4.3 McDonald's omega is calculated from a one-factor model, so each item can have its own loading instead of the equal loadings that alpha assumes (tau-equivalence). When the loadings are similar, omega and alpha agree, as they do for the social subscale (both 0.75); when the loadings differ, alpha underestimates reliability and omega is the better estimate, as for the emotional subscale (loadings 0.83, 0.20 and 0.58; alpha 0.51 and omega 0.57). If a question shows omega noticeably above alpha, look for one item whose loading is much weaker than the others. When a published scale has a weak item, resist the urge to delete it just because alpha would rise: keep the published scale as the main measure so that your results remain comparable with other studies, report alpha and omega, name the weak item, and run a sensitivity analysis without it, because an alpha chosen after looking at the data overstates reliability in new samples. If the conclusions agree, the weak item has not driven them.
Knowledge check: this section

1. Fifty patients with a stable chronic illness complete a fatigue questionnaire (scored 0 to 40) twice, two weeks apart. The Pearson correlation between the two occasions is 0.86, the intraclass correlation for absolute agreement is 0.68, and the mean score is 3 points higher on the second occasion. Which statement is correct?

Internal consistency asks whether items move together within one administration. Here the same people answer the same measure on two occasions, which is the design for test-retest reliability, summarized with an intraclass correlation.
Giving a measure twice to people whose condition should be stable estimates test-retest reliability. On Koo and Li's bands 0.68 is moderate, and the intraclass correlation for absolute agreement counts the shared rise in scores as disagreement, whereas the Pearson correlation ignores it.
The design is identified correctly, but the statistic is not. A Pearson correlation ignores a shift that affects everyone, so it cannot measure agreement; the intraclass correlation for absolute agreement does, and its value of 0.68 is moderate.
Inter-rater reliability needs two or more raters assessing the same people independently. Here the patients rate themselves on two occasions, which is the design for test-retest reliability.
The correct answer is b. The information needed is the design (the same people answer the same measure on two occasions while their condition should not change), the intraclass correlation and the shift in the mean. Giving a measure twice estimates test-retest reliability, and Koo and Li read an intraclass correlation of 0.50 to 0.75 as moderate, so 0.68 is moderate. The intraclass correlation for absolute agreement counts a shift that affects everyone as disagreement, whereas a Pearson correlation ignores it, which is why option c is wrong: the patients did not receive the same score twice, and a systematic rise while the illness is stable means that the scores depend partly on the occasion. (Section 2, 3.2 Three Kinds of Reliability, Table 7.1, and 3.3.1 The Intraclass Correlation for Test-Retest Reliability, Worked Example 7.1; Objectives 3.2 and 3.3)

2. A six-item health anxiety scale is answered from 1 to 5. Item H4, "I rarely worry about my health", is worded in the opposite direction to the other five items, and the scoring manual says that it must be reverse-coded. An analyst runs alpha() on the items as stored and gets the output below. A colleague proposes dropping H4, because alpha would rise from 0.43 to 0.74 without it. What should be done?

Some items ( H4 ) were negatively
correlated with the first principal
component and probably should be
reversed.

 raw_alpha
      0.43

 Reliability if an item is dropped:
   raw_alpha
H1      0.22
H2      0.23
H3      0.24
H4      0.74
H5      0.25
H6      0.26

 Item statistics
   r.drop
H1   0.45
H2   0.43
H3   0.42
H4  -0.56
H5   0.41
H6   0.39
This treats a coding error as a bad item. H4 is worded in the opposite direction and the manual says it must be reversed, so its negative r.drop reflects a missed reversal; recoding it restores the published scale and also raises alpha.
check.keys = TRUE flips items from the numbers alone, without reading their wording, so it can flip an item that was coded correctly. The scoring manual decides which items are reversed, and it names H4.
A negative r.drop for an oppositely worded item is the signature of a missed reversal, not of a separate construct. The manual requires H4 to be reverse-coded, after which it should move with the other items.
The negative r.drop shows that H4 moves opposite to the other items, which fits its opposite wording when it has not been reversed. The manual's rule recodes it, and alpha is then recomputed on the correctly scored items.
The correct answer is d. The information needed is the wording of H4, the scoring manual and the sign of its r.drop, the correlation of the item with the sum of the other items. A negative r.drop means that people who score high on the other items score low on H4, which is what an oppositely worded item does when it has not been reversed, and psych's warning points the same way. The remedy is the manual's: recode H4 as (5 + 1) − code and recompute alpha; in data like these alpha then rises to about 0.78, above the 0.74 from dropping the item, and the published scale stays intact. Option a treats a coding error as a bad item, and option b lets the software decide which items to reverse, which can flip a correctly coded item. (Section 2, 2.3 Spotting a Missed Reversal: The Scoring Manual Decides, and Activity 7.2; Objective 2.3)

3. A 20-item scale has an alpha of 0.88 in a survey of 2,000 adults, and the average correlation between pairs of items is 0.27. Which conclusion is justified?

An alpha of 0.88 meets Nunnally's guide for research comparing groups but falls short of the 0.90 or more for individual decisions. Alpha rises with the number of items, so with 20 items it cannot show that they measure one construct.
A high alpha does not establish unidimensionality. Alpha rises with the number of items, so 20 items in two related clusters can still give a high value; the number of constructs is a question for factor analysis.
Concern about redundant items starts above about 0.95, and an average inter-item correlation of 0.27 is modest. The high alpha here comes mainly from the number of items, not from items that repeat each other.
This applies the guide for decisions about individuals to group research. Nunnally's guides put about 0.80 as adequate for basic research comparing groups, so 0.88 is adequate for that use.
The correct answer is a. Two pieces of information matter: the value of alpha, to compare with Nunnally's guides (about 0.70 for early research, about 0.80 for basic research comparing groups, 0.90 or more for decisions about individuals), and the number of items. The standardized formula (Equation 7.6) explains why the value is high: alpha rises with the number of items, so with 20 items even a modest average correlation of 0.27 gives an alpha of 0.88, and items from two related clusters can do the same. Option b is the tempting error: whether the items measure one construct is a question for factor analysis. Option c is wrong because concern about redundant items starts above about 0.95. (Section 2, 4.1 Cronbach's Alpha and the Item Statistics, Equation 7.6, and 4.2 What Alpha Assumes and What It Cannot Show; Objectives 4.1 and 4.2)

4. A published three-item emotional support subscale is analyzed in a new sample. Alpha is 0.58 and McDonald's omega is 0.63. The one-factor loadings are 0.80, 0.25 and 0.70. Item S2 has an r.drop of 0.21, against 0.50 and 0.47 for the other two items, and alpha would rise to 0.72 if S2 were dropped. Which plan is best?

Dropping a published item after looking at the data makes the scores incomparable with other studies and tends to overstate reliability in new samples. The item is kept and reported, and a sensitivity analysis without it shows whether the result depends on it.
Omega is more appropriate than alpha when loadings are unequal, but it is still below 0.70 here, so the subscale has low internal consistency. Both values should be reported, and the weak item should be named.
With unequal loadings alpha underestimates the reliability of the sum, so omega is higher. The published items are kept as the main measure, S2 is reported as weak, and a sensitivity analysis checks whether the result depends on it.
An unreversed item has a negative r.drop. S2 has a low but positive r.drop and a low loading, which marks a weak item that is scored in the right direction, so reverse-coding it would introduce an error.
The correct answer is c. The information needed is the pattern of loadings and the item statistics. Alpha assumes equal loadings (tau-equivalence); with loadings of 0.80, 0.25 and 0.70 that assumption fails, so alpha underestimates the reliability of the sum, and omega, which gives each item its own loading, is higher. S2 has a low but positive r.drop, which marks a weak item rather than an unreversed one (that would be negative), so option d is wrong. Option a is tempting, but dropping a published item after looking at the data makes the scores incomparable with other studies and overstates reliability in new samples, so the item is kept, reported and tested with a sensitivity analysis; and because both values are below 0.70, the subscale is described as having low internal consistency (option b). (Section 2, 4.3 McDonald's Omega and What to Do About a Weak Item, Equation 7.7 and the running case card; Objectives 4.3 and 4.1)

5. An analyst computes alpha = 0.34 for the five components of the Steptoe social isolation index in the CSCS (being unmarried and not living with a partner; less than monthly contact with children, with other family and with friends; and taking part in no groups) and writes that "the index is unreliable and should be replaced." Which response is best?

The 0.70 guide applies to reflective scales, in which one construct causes every answer. The Steptoe components define social isolation and need not correlate, so alpha is not a measure of the index's quality.
Being unmarried, rarely seeing family or friends and belonging to no groups make up isolation, so the index is formative. Alpha assumes a reflective model, so a low value is expected and says nothing about the index.
Omega relaxes alpha's assumption of equal loadings, but it still rests on a one-factor reflective model, so it is equally out of place for a formative index. An index is judged by its coverage of the construct and its relations with other variables.
Adding components to raise alpha treats the index as a reflective scale. Formative components are chosen to cover the definition of the construct, and their correlations, and hence alpha, are not a guide to the index's quality.
The correct answer is b. The information needed is how the components relate to the construct: a person is isolated because they are unmarried, rarely see family or friends and belong to no groups, so the components define isolation (formative measurement) rather than being caused by it. Alpha assumes a reflective model in which one construct causes every answer, so components that barely correlate (−0.10 to 0.36 in the CSCS) give a low alpha that says nothing about the quality of the index. Option c is tempting, but omega also rests on a one-factor model, so it is equally out of place; an index is judged instead by whether its components cover the definition of the construct and whether it relates to other variables as expected. (Section 2, 1.3 Internal Consistency Applies Only to Scales, and Activity 7.2; Objective 1.3)

6. A health region's online survey asks a five-item vaccine confidence scale (scored 5 to 25) once. In a sub-study, 45 parents answer the scale again three weeks later, and a local measles outbreak is reported between the two occasions. The intraclass correlation for absolute agreement is 0.71, the Pearson correlation is 0.84, and the mean score rises from 17.6 to 19.0. Which statement is correct?

The sub-study gives the same scale to the same parents on two occasions, which is the design for test-retest reliability. Internal consistency comes from a single administration and does not describe stability over time.
Stability over time needs a second administration. A survey that asks the items once, however large, supports only internal consistency, which is why the sub-study was needed.
A Pearson correlation ignores a shift that is the same for everyone, so it describes ordering rather than agreement. The intraclass correlation for absolute agreement is the statistic for test-retest reliability, and here the rise may partly reflect real change after the outbreak.
Only the repeated administration can estimate test-retest reliability, and 0.71 is moderate on Koo and Li's bands. The method assumes the construct did not change; an outbreak may have shifted real confidence, so part of the rise may be change rather than error.
The correct answer is d. The information needed is the design (the same parents answer the same scale on two occasions), the intraclass correlation and what happened between the occasions. Giving a measure twice estimates test-retest reliability, whereas a survey that asks the items once supports only internal consistency, which is why options a and b are wrong. On Koo and Li's bands (0.50 to 0.75 moderate), 0.71 is moderate. The intraclass correlation for absolute agreement counts a rise that affects everyone as disagreement, and test-retest reliability assumes that the construct did not change between the occasions; an outbreak can change real confidence, so part of the rise may be real change rather than error. Option c is the tempting error: a Pearson correlation ignores a shift that is the same for everyone, so it is not the statistic for agreement. (Section 2, 3.2 Three Kinds of Reliability, Table 7.1 and the callout "What the CSCS Can and Cannot Show", and 3.3.1 The Intraclass Correlation for Test-Retest Reliability, Worked Example 7.1; Objectives 3.2 and 3.3)
● Complete the quiz to continue.
Section 3 of 4

Validity

⏱ Estimated time: 40 minutes
Lesson 7 · Section 3

Validity

Do the scores mean what we claim they mean, for this use and in this population?

Reliability and validity

Consistent is different from correct

Tight and centred

The measure is reliable and valid.

Tight and off target

The measure is reliable and measures the wrong thing.

Scattered

The measure is unreliable, which also limits its validity.

Reliability is necessary for validity, and it is never enough to establish it.

Validity as an argument

Valid for what, and for whom?

Validity is an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment.Messick (1989, p. 13)

The argument names the intended use and population, then gathers evidence for each claim.

Kinds of evidence

Four families of validity evidence

Content and face

The items cover the construct, as judged by experts and by respondents.

Criterion

Scores agree with a gold standard now (concurrent) or predict a later outcome (predictive).

Construct

Scores relate to other variables as theory predicts (convergent, discriminant and known groups).

Structural

The items group into the domains the theory describes (Section 4).

Convergent and discriminant evidence

Correlations in 3,053 CSCS participants

MeasureTotalSocial subscaleEmotional subscale
UCLA Loneliness Scale (3 items)0.450.320.39
Perceived social support (MSPSS)−0.49−0.48−0.26
Anxiety symptoms (GAD-2)0.330.210.32

Social loneliness tracks social support more closely, and emotional loneliness tracks anxiety and the UCLA scale more closely.

Known groups

Scores fall as the number of close friends rises

Grouped bar chart of mean social and emotional loneliness subscale scores by number of close friends. Social loneliness falls from 2.47 with no close friends to 1.13 with five or more; emotional loneliness falls from 2.45 to 1.74.
Mean subscale scores (0 to 3) in the 2021 CSCS wave.
COSMIN taxonomy

Three domains of measurement properties

Reliability

This domain covers internal consistency, test-retest and inter-rater reliability, and measurement error.

Validity

This domain covers content (and face), criterion and construct validity, including structural validity, hypothesis testing and cross-cultural validity.

Responsiveness

This domain covers the ability of scores to detect change over time.

Interpretability is listed as an important characteristic beside the three domains.

Population matters

Does the scale work the same way in every group?

  • Measurement invariance asks whether the items relate to the construct in the same way across groups.
  • Without it, a difference in mean scores may arise from how the items are read in each group.
  • It is tested with the confirmatory factor analysis introduced in Section 4.
Carry forward

What to take into the next section

  • Validity is an argument for a use of scores in a population, built from several kinds of evidence.
  • The COSMIN taxonomy organizes the evidence into reliability, validity and responsiveness.
  • CSCS correlations and known groups support the scale and suggest two distinct subscales.

Overview

A reliable score is consistent, and consistency does not show that the score measures the intended construct. Validity concerns the meaning of scores: whether a loneliness score reflects loneliness or instead reflects low mood, social desirability or something else that the items happen to share. Texts written for readers appraising a study often use construct validity for the broad question of whether an instrument captures its construct, which this lesson calls validity. This lesson reserves construct validity for evidence that scores relate to other measures as theory predicts (convergent, discriminant and known-groups evidence), and the COSMIN taxonomy groups structural and cross-cultural validity under construct validity. This section takes the analyst's perspective. It separates validity from reliability, classifies the kinds of validity evidence, frames validity as an argument for a particular use of scores in a particular population, organizes the evidence with the COSMIN taxonomy, explains how to judge whether a change in scores is important, gathers construct validity evidence for the De Jong Gierveld scale from the CSCS data, and explains why comparisons between groups depend on measurement invariance.

Learning Objectives

  • 5.1 Distinguish reliability from validity, and explain why a reliable measure is not necessarily valid.
  • 5.2 Classify validity evidence as content and face validity, criterion validity (concurrent and predictive), construct validity (convergent, discriminant and known groups) or structural validity.
  • 5.3 Explain validity as an argument assembled from several kinds of evidence for a particular use in a particular population, and why a scale should not simply be called "validated".
  • 5.4 Use the COSMIN taxonomy to organize the measurement properties of an instrument.
  • 5.5 Define the minimal important difference, compare anchor-based and distribution-based estimates of it, and judge whether a given difference is important.
  • 5.6 Interpret convergent, discriminant and known-groups evidence computed in R.
  • 5.7 Explain why a comparison of mean scores between groups depends on measurement invariance.

5.1 Reliability and Validity

Reliability concerns random error, the scatter of scores around a person's true score. Validity concerns systematic error, whether the true score itself corresponds to the intended construct. The two are linked in one direction only. A score that is mostly random error cannot correlate strongly with anything, so low reliability limits validity. A highly reliable score can still measure the wrong construct, as a bathroom scale that always reads two kilograms heavy is consistent and wrong, and a loneliness scale could be highly reliable and still mainly measure general distress. Reliability is therefore necessary for validity but never enough to establish it, and a high alpha is never by itself evidence that a measure is valid.

Figure 7.4 illustrates the relation with three dartboards, in which the spread of the darts stands for reliability and their position relative to the bullseye stands for validity.

High reliabilityHigh validity High reliabilityLow validity Low reliabilityLow validity
Figure 7.4. Reliability is how tightly the darts cluster; validity is whether they cluster on the bullseye. The middle board shows a consistent measure of the wrong thing.

A consistent score can therefore still miss its target, and validity has to be supported by evidence of its own. The next subsection describes the kinds of evidence from which that support is built.

5.2 Kinds of Validity Evidence

Validity evidence is usually grouped into four families, each of which supports a different claim about the scores. Content evidence (with face validity as its weakest form) asks whether the items cover the construct. Criterion evidence asks whether scores agree with a gold standard measured at the same time (concurrent) or later (predictive). Construct evidence asks whether scores relate to other variables as theory predicts, through convergent, discriminant and known-groups comparisons. Structural evidence asks whether the items form the domains the theory describes.

The six cards below define each kind of evidence and give an example from the loneliness scale or from screening. Click each card, and note in particular how convergent and discriminant evidence are judged together and which kind of evidence the reviewer's first question calls for.

Content ValidityClick to explore
Face ValidityClick to explore
Criterion ValidityClick to explore
Convergent and Discriminant EvidenceClick to explore
Known-Groups ValidityClick to explore
Structural ValidityClick to explore

Criterion validity is the most direct evidence when a trusted gold standard exists, as it does for many diagnostic tests. For psychological constructs such as loneliness there is no gold standard, so the argument relies mainly on content, construct and structural evidence (Cronbach & Meehl, 1955). Construct validity is tested by writing down, before looking at the data, the correlations and group differences that theory predicts, and then checking them. COSMIN calls this hypothesis testing. Stating the hypotheses in advance guards against reading support into whatever pattern appears.

When a score is used with a cut-point to classify people, criterion validity evidence is summarized by comparing the classification with the gold standard. The sensitivity is the share of people who have the condition according to the gold standard whom the cut-point flags, and the specificity is the share of people without the condition whom it correctly does not flag. For example, a sensitivity of 0.70 and a specificity of 0.90 mean that the cut-point finds about 7 in 10 people with the condition and misses about 3 in 10, while it wrongly flags about 1 in 10 people without the condition.

Each kind of evidence supports one claim about the scores. The next subsection describes how current measurement theory combines them into a single judgement.

5.3 Validity as an Argument

Early textbooks listed "types of validity" as if a scale either had or lacked each one. Current measurement theory treats validity as a single, overall judgement about how well the evidence supports a particular interpretation and use of scores (Messick, 1989, 1995; AERA, APA & NCME, 2014). By this we mean that validity belongs to the way scores are interpreted and used, not to the questionnaire itself. The same loneliness scale might be well supported for comparing average loneliness between groups in a survey and poorly supported for deciding which individual patients should be referred to a program, because these are different uses that make different claims about what the scores mean. The different kinds of evidence in 5.2 are therefore not separate validities that a scale collects; they are pieces of support for one interpretation. Kane (2013) describes how these pieces are put together, calling validation the construction of an argument in two steps. The first step states the interpretation and use: for example, "scores on the De Jong Gierveld scale rank adults in an online Canadian survey by their degree of loneliness, and differences in mean scores between groups reflect differences in loneliness". The second step gathers evidence for each claim that this interpretation depends on, including that the items cover loneliness, that the answers form the intended domains, that the scores relate to other variables as expected, and that the items work the same way in the groups being compared.

Two consequences follow. Support for a scale is always relative to a stated use in a stated population, so a scale should not simply be called "valid" or "validated" in general. Evidence that the De Jong Gierveld scale works well in the populations in which it was developed supports, without settling, its use with Canadian adults of all ages in an online survey, and evidence gathered for comparing groups in research does not transfer automatically to decisions about individuals. Validation is also cumulative: each study that uses a scale and reports how its scores behave adds to the argument. A Measures paragraph therefore states what the evidence supports, for which use and in which population, and what is missing.

An argument assembled from many kinds of evidence needs a way to organize them, and the next subsection presents the taxonomy most often used for that purpose.

5.4 The COSMIN Taxonomy

The COSMIN initiative (Consensus-based Standards for the selection of health Measurement Instruments) developed, through an international Delphi study, a taxonomy and definitions for the measurement properties of health instruments (Mokkink et al., 2010). It groups the properties into three domains: reliability, validity and responsiveness. It is widely used in systematic reviews of measurement instruments, and it is a useful checklist for the Measures paragraph of a manuscript.

Table 7.2 lists the three COSMIN domains with their measurement properties, the question that each property answers, and the evidence for the De Jong Gierveld scale that this lesson provides. Read the last column as a checklist: it shows which properties the CSCS data can assess and which must come from published studies or a different design.

Table 7.2. The COSMIN domains and measurement properties, with the evidence for the De Jong Gierveld scale in this lesson.

DomainMeasurement propertyQuestion it answersEvidence for the De Jong Gierveld scale in this lesson
ReliabilityInternal consistencyDo the items of each domain move together?Alpha and omega in the CSCS sample (Section 2)
ReliabilityAre scores stable over occasions or raters?Published studies only; not estimable from one CSCS wave
Measurement errorHow large is the error in a person's score?Not assessed in this lesson
ValidityContent validity (including face validity)Do the items cover the construct?Item wording based on Weiss's two domains (Section 1)
Construct validity: structural validity, hypothesis testing, cross-cultural validityDo the items form the predicted domains, and do scores relate to other variables as predicted?Factor analysis (Section 4); correlations and known groups (this section)
Criterion validityDo scores agree with a gold standard?No gold standard for loneliness
ResponsivenessResponsivenessDo scores change when the construct changes?Needs longitudinal data

COSMIN also describes interpretability (for example, what size of difference in scores is meaningful) as an important characteristic of an instrument, alongside the three domains.

Table 7.2 shows that, beyond internal consistency, the CSCS data can add two kinds of validity evidence for the De Jong Gierveld scale: hypothesis testing through correlations and known groups, which 5.6 computes, and structural validity through factor analysis, which Section 4 tests. Before turning to those, the next subsection takes up the property that sits beside the three domains, interpretability.

5.5 The Minimal Important Difference

Responsiveness asks whether scores change when the construct changes, and interpretability asks what a change of a given size means. Statistical significance shows whether a difference is likely to be real, not whether it matters. The minimal important difference (MID) is the smallest change in scores that people would regard as important, or that would justify a change in care (Jaeschke, Singer & Guyatt, 1989). It is estimated in two ways. An anchor-based estimate links score changes to an external judgement, the anchor: for example, participants report at follow-up whether their loneliness is "a little better", and the MID is the mean change in score among those who chose that answer. A distribution-based estimate uses only the spread and the reliability of the scores. Two common versions are half a standard deviation of the baseline scores (Norman, Sloan & Wyrwich, 2003) and one standard error of measurement (Wyrwich, Tierney & Wolinsky, 1999). Equation 7.8 defines the standard error of measurement.

Standard error of measurement
\[ \text{SEM} = SD \times \sqrt{1 - \text{reliability}} \]Eq 7.8
The standard error of measurement is the standard deviation of the scores multiplied by the square root of one minus their reliability. It describes the typical size of the error in one person's score.

The two approaches answer different questions. Distribution-based values describe how large a change is compared with the spread of scores and the error in them. The SEM, for example, tells us how far a person's observed score typically sits from their true score, so a change smaller than one SEM could easily be produced by measurement error alone. However, these values say nothing directly about whether people would notice or value the change, because no one has been asked. Anchor-based estimates, by contrast, tie the change to people's own judgement of whether they are better. For this reason anchor-based estimates are preferred when they exist, and distribution-based values serve as a check. As an illustration, consider a loneliness scale scored 0 to 6 whose baseline scores have a standard deviation of 1.6 and a reliability of 0.75 (hypothetical values). In R, half the standard deviation (1.6 / 2) and the SEM (1.6 * sqrt(1 - 0.75)) both return 0.8 points; the two coincide whenever the reliability is 0.75. If participants who reported feeling "a little better" improved by 1.0 point on average, the anchor-based MID would be 1.0. A program that lowers the mean score by 0.4 points then produces a change below every estimate of the MID, even if a large trial finds it statistically significant.

A difference is therefore judged against every estimate of the MID that is available, with the anchor-based value carrying most weight. The next subsection returns to the CSCS and gathers the construct validity evidence that Table 7.2 lists for the De Jong Gierveld scale.

5.6 Construct Validity Evidence from the CSCS

Before computing anything, Amira writes down what the theory of the scale predicts. Loneliness is usually defined as the distress that follows from a gap between the relationships a person has and those they want (Perlman & Peplau, 1981), and Weiss (1973) distinguished its emotional and social forms. The total score should therefore correlate positively with another loneliness measure, the three-item UCLA Loneliness Scale (Hughes et al., 2004), and negatively with perceived social support, measured in the CSCS with the Multidimensional Scale of Perceived Social Support (Zimet et al., 1988). It should correlate less strongly with anxiety symptoms (measured with the two-item GAD-2), a related but distinct construct. The social subscale, which concerns the wider network, should relate more strongly to social support and to the number of close friends than the emotional subscale does.

Activity 7.3, at the end of this section, tests these predictions with a correlation matrix for the convergent and discriminant hypotheses and a comparison of subscale means by number of close friends for the known-groups hypothesis. The results support most of the predictions. The total score correlates 0.45 with the UCLA scale and −0.49 with perceived social support (convergent evidence; the sign is negative because more support goes with less loneliness), and only 0.33 with anxiety (discriminant evidence), so the convergent correlations exceed the discriminant one in absolute size. The gap is modest, which is common for constructs that share an emotional component, and a discriminant correlation needs only to be clearly lower than the convergent ones, not zero. The two subscales show the distinct patterns the theory predicts: the social subscale relates most strongly to social support (−0.48, compared with −0.26 for the emotional subscale), whereas the emotional subscale relates more strongly to the UCLA scale and to anxiety. The two subscales correlate only 0.21 with each other.

Figure 7.5 plots the known-groups comparison for the two subscales. Both fall steadily as the number of close friends rises, and the gradient is much steeper for social loneliness (2.47 to 1.13) than for emotional loneliness (2.45 to 1.74), as Weiss's distinction predicts.

Grouped bar chart of mean social and emotional loneliness subscale scores by number of close friends: none, 1 to 2, 3 to 4, and 5 or more. Social loneliness falls from 2.47 to 1.13; emotional loneliness falls from 2.45 to 1.74.
Figure 7.5. Known-groups evidence in the 2021 CSCS wave. Both subscales fall as the number of close friends rises, and the gradient is steeper for social loneliness, as Weiss's distinction predicts.

These results are encouraging, but two features of the data limit how far they can be pushed, as the caution below explains.

Interpreting Validity Correlations with Care

Validity correlations are themselves attenuated by unreliability (Section 1). The emotional subscale has an alpha of 0.51, so its correlations with other measures are weakened, and part of the difference between the subscales' correlations could reflect their different reliabilities. The CSCS is also a cross-sectional online survey that recruited volunteers, so these correlations describe this sample and support, without proving, the same pattern in other populations.

Figure 7.5 compares mean scores between groups, and any such comparison rests on an assumption that the next subsection examines.

5.7 Measurement Invariance

A comparison of mean scores between groups, such as older and younger adults, assumes that the items relate to the construct in the same way in each group. If younger adults interpret "I miss having people around me" differently from older adults, a difference in mean scores could arise from the item's wording rather than from a real difference in loneliness. The same concern applies to a scale answered in two languages, where a translated item may be understood differently; COSMIN places this cross-cultural validity under construct validity (Table 7.2). Measurement invariance is the property that the items measure the construct in the same way in every group being compared. By this we mean that two people in different groups who have the same true level of loneliness should be expected to give the same answers to each item, so that any difference in mean scores between the groups reflects a difference in loneliness and not a difference in how the items are understood. Measurement invariance testing checks this assumption by fitting the confirmatory factor model of Section 4 in each group and testing whether the loadings and item intercepts are equal. Equal loadings mean that each item is tied to the construct equally strongly in every group. Equal intercepts mean that, at the same level of the construct, people in each group tend to give the same answer to the item, which is what a comparison of means requires. A full treatment is beyond this lesson. In a manuscript that compares groups, it is good practice to state that the comparison assumes invariance, to cite any published invariance evidence and to note comparisons for which it is missing.

Activity 7.3 gathers the construct validity evidence of 5.6 in R and ends with a question on what must be true of the scale before mean scores are compared between groups. Run the script alongside the activity and answer its questions from your own output.

R Activity 7.3: convergent, discriminant and known-groups evidence

Text version of the activity: the code of the run-along script, its console output and notes on reading it.

load(url("https://raw.githubusercontent.com/jorgeandr3s/heal/main/cscs/public_data/CSCS2025_full_cleaned_deidentified_data_and_metadata.RData"))   # CSCS data, read from the web
data <- data[data$SURVEY_collection_year == 2021, ]   # one row per person
nrow(data)
Console output
[1] 4045

The file is read straight from the web (an internet connection is needed). Keeping the 2021 wave leaves 4,045 people, one row per person.

social    <- data$LONELY_dejong_social_loneliness_sub_scale_score      # 0 to 3
emotional <- data$LONELY_dejong_emotional_loneliness_sub_scale_score   # 0 to 3
val <- data.frame(total     = data$LONELY_dejong_emotional_social_loneliness_scale_score,
                  social    = social,
                  emotional = emotional,
                  ucla      = data$LONELY_ucla_loneliness_scale_score,
                  support   = data$PSYCH_zimet_multidimensional_social_support_scale_score,
                  anxiety   = data$WELLNESS_gad_score)
val <- na.omit(val)          # people with every score
nrow(val)
Console output
[1] 3053

The De Jong Gierveld subscale scores (0 to 3) and total score (0 to 6) supplied with the CSCS file are the same, person by person, as the scores built with the published rule in Section 1. The other measures were scored by the CSCS team and are used as supplied: the three-item UCLA Loneliness Scale (3 to 9), the Multidimensional Scale of Perceived Social Support (an average from 1 to 7) and the two-item GAD-2 anxiety screener (0 to 6). The 3,053 people in val have every score.

round(cor(val), 2)           # correlations between every pair of measures
Console output
total social emotional ucla support anxiety total 1.00 0.83 0.72 0.45 -0.49 0.33 social 0.83 1.00 0.21 0.32 -0.48 0.21 emotional 0.72 0.21 1.00 0.39 -0.26 0.32 ucla 0.45 0.32 0.39 1.00 -0.41 0.43 support -0.49 -0.48 -0.26 -0.41 1.00 -0.30 anxiety 0.33 0.21 0.32 0.43 -0.30 1.00

Reading the matrix. Read down the total, social and emotional columns (the matrix is symmetric, so the rows give the same values). The convergent correlations for the total are 0.45 with the UCLA scale and −0.49 with perceived social support (negative because more support goes with less loneliness), and the discriminant correlation with anxiety is lower at 0.33. The social subscale correlates −0.48 with support and 0.21 with anxiety. The emotional subscale correlates −0.26 with support and 0.32 with anxiety. The two subscales correlate 0.21 with each other.

friends <- data$CONNECTION_social_num_close_friends_grouped
friends[friends == "Presented but no response"] <- NA
friends <- factor(friends, levels = c("None", "1–2", "3–4", "5 or more"))
table(friends)               # people in each group
round(tapply(social, friends, mean, na.rm = TRUE), 2)   # mean social score
Console output
friends None 1–2 3–4 5 or more 179 1309 1435 681 None 1–2 3–4 5 or more 2.47 2.08 1.78 1.13

The grouping variable is set as a factor with its levels in order, so that the groups print from fewest to most friends, and the 20 people who saw the question without answering it become NA. The smallest group, people with no close friends, has 179 people. The last line gives the mean social subscale score (0 to 3) in each group: social loneliness falls from 2.47 among people with no close friends to 1.13 among those with five or more. Question 3 asks you to run the same line with emotional in place of social.

Files for this activity Data Code Answer key

R Reflect on what you just ran

Answer each question from your own output, as on the final exam: name the kind of evidence, say whether it supports the scale, and check it against the prediction made before the analysis.

1. Using the correlations of the total score with the UCLA Loneliness Scale, perceived social support and anxiety symptoms, name the kind of validity evidence that each correlation provides, and say whether the pattern supports the scale.

Model answerThe correlation of 0.45 with the UCLA Loneliness Scale is convergent (construct) validity evidence, because the UCLA scale is another measure of loneliness. The correlation of −0.49 with perceived social support, a closely related construct, is also convergent evidence; it is negative because more support goes with less loneliness, as predicted, so it supports the scale rather than counting against it. The correlation of 0.33 with anxiety symptoms is discriminant evidence: anxiety is related to loneliness but distinct from it, so its correlation should be clearly lower than the convergent ones, although it need not be zero. The pattern supports the scale, because the discriminant correlation is lower than both convergent correlations in absolute value; the gap is modest, though, and a correlation of 0.45 with another loneliness scale is only moderate.

2. Before the analysis, Amira predicted that the social subscale would correlate more strongly with perceived social support than the emotional subscale does. Report the two correlations and say whether they support the prediction. Why should the comparison be read with some caution?

Model answerThe social subscale correlates −0.48 with perceived social support and the emotional subscale −0.26, so the social subscale's association is clearly stronger and the prediction is supported. Because the prediction was written down before the data were analyzed, this is the hypothesis testing that COSMIN describes, and it is construct validity evidence that the two subscales capture different domains of loneliness. The comparison should be read with some caution, because the emotional subscale is less reliable (an alpha of 0.51 in Section 2) and random measurement error weakens (attenuates) correlations toward zero, so part of the difference could reflect the different reliabilities of the two subscales.

3. Run your own code: at the end of the script, copy line 31, change social to emotional and run it. Name the kind of validity evidence that the gradients of the two subscales provide, and say whether they support the subscales. Why does the difference in the size of the two gradients fit Weiss's distinction between social and emotional loneliness? Before a report compares mean scores between groups, what must be true of the scale?

Model answerThe emotional subscale means are 2.45, 2.39, 2.21 and 1.74, from no close friends to five or more. Both gradients are known-groups (construct) validity evidence: people with fewer close friends, whom theory expects to be lonelier, score higher, in the direction predicted before the analysis, so both support the subscales. Social loneliness falls steeply, from 2.47 among people with no close friends to 1.13 among those with five or more, whereas emotional loneliness falls much less, from 2.45 to 1.74; Weiss described social loneliness as the absence of a wider network and emotional loneliness as the absence of a close attachment, so the number of friends should matter more for social loneliness, which is what the data show. Comparing mean scores between groups assumes measurement invariance: the items must relate to loneliness in the same way in each group, otherwise a difference in means may reflect how the items work in each group rather than a real difference in loneliness (it is checked with a confirmatory factor model in each group, or with published evidence).
Saved.

Your output from Activity 7.3 reproduces the correlations and the known-groups gradients interpreted in 5.6. With the kinds of evidence, the COSMIN headings and these results in place, the task below shows how they are assembled into a validity argument for a Measures paragraph, drawing on a measure's published sources and on an analysis of the CSCS data.

📋 Try it: assemble the validity evidence for a CSCS measure

Choose one multi-item measure in the CSCS (for example, the De Jong Gierveld scale, the three-item UCLA Loneliness Scale, the Multidimensional Scale of Perceived Social Support, the PHQ-2 or the GAD-2). First, state the interpretation and use in one sentence, naming the population. Second, find the scale's original development paper and, if one exists, a validation study in a population like the CSCS sample, and note the evidence each provides under the COSMIN headings (content, structural, hypothesis testing, criterion, reliability). Third, note what an analysis of the CSCS data adds: internal consistency in the study sample, and one or two pre-stated correlations or group differences. Fourth, record what is missing, such as test-retest evidence or measurement invariance across the groups being compared. The result is a table of four rows that becomes two or three sentences of a Measures paragraph like the one drafted in Section 4 (Worked Example 7.2).

This section has separated validity from reliability, classified the kinds of validity evidence, treated validity as an argument for a particular use of scores in a particular population, organized the evidence with the COSMIN taxonomy, and gathered construct validity evidence for the De Jong Gierveld scale from the CSCS. Structural validity, the evidence that the reviewer's first question calls for, requires a latent variable model, and Section 4 introduces the models used to test it. The key takeaways below draw the section together before its knowledge check.

Key Takeaways

  • 5.1 Reliability and validity answer different questions, and keeping them apart is one of the most useful habits in this lesson. Reliability concerns random error, the scatter of scores around each person's true score, whereas validity concerns systematic error, that is, whether the score measures the intended construct for its stated use. A score can be consistent and consistently wrong, like a bathroom scale that always reads 2 kg heavy, so reliability is necessary for validity and never enough to establish it. The link runs one way only: a score that is mostly random error cannot correlate strongly with anything, so low reliability limits validity, but a highly reliable loneliness score could still mainly measure general distress. When a question claims that a measure is valid because its alpha is high, explain that alpha shows consistency only and that validity needs evidence of its own.
  • 5.2 Each kind of validity evidence supports a different claim, so the skill on an exam is to match the evidence in front of you to the question it answers. Content evidence asks whether the items cover all of the construct and nothing else, usually as judged by experts, and face validity, the weakest kind, asks whether the items look relevant to the people answering them. Criterion evidence compares scores with a gold standard measured at the same time (concurrent) or later (predictive); for a cut-point it is summarized with the sensitivity, the share of people with the condition whom the cut-point flags, and the specificity, the share of people without it whom the cut-point correctly does not flag. Construct evidence asks whether scores relate to other variables as theory predicts: a strong correlation with a measure of the same or a closely related construct is convergent evidence, a clearly weaker correlation with a related but distinct construct is discriminant evidence, and a difference in the predicted direction between groups expected to differ is known-groups evidence. Structural evidence asks whether the items form the intended domains, and it comes from factor analysis or item response theory. For constructs such as loneliness there is no gold standard, so the argument rests mainly on content, construct and structural evidence.
  • 5.3 Current measurement theory does not treat validity as a stamp that a scale either has or lacks. Validity is an argument for a stated use in a stated population, built from content, criterion, construct and structural evidence. You build it in two steps: state the interpretation and use (for example, that scores rank adults in an online Canadian survey by their degree of loneliness), then gather evidence for each claim on which that interpretation depends, and the argument keeps growing as each new study reports how the scores behave. That is why a scale should not simply be called "validated": evidence from the population in which a scale was developed supports, without settling, its use in a different population or for a different purpose. When a question asks you to rewrite "a validated scale", state what the evidence supports, for which use and in which population, and say what is missing, such as test-retest evidence when the data come from a single survey.
  • 5.4 The COSMIN taxonomy gives you a ready-made set of headings for organizing everything that is known about an instrument. It groups measurement properties into three domains: reliability (internal consistency, reliability over occasions or raters, and measurement error), validity (content validity including face validity, criterion validity, and construct validity, which covers structural validity, hypothesis testing and cross-cultural validity) and responsiveness (whether scores change when the construct changes), with interpretability, such as what size of change matters, beside them. Use it as a checklist: for each property, note whether the evidence comes from published studies or from your own data, and which properties your design cannot assess. For the De Jong Gierveld scale in the CSCS, for example, internal consistency, structural validity and hypothesis testing can be assessed from the data, whereas stability over time must come from published studies, there is no gold standard for criterion validity, and responsiveness needs longitudinal data.
  • 5.5 Statistical significance tells you whether a difference is likely to be real, not whether it matters, and the minimal important difference (MID) fills that gap: it is the smallest change in scores that people would regard as important. An anchor-based estimate links score changes to an outside judgement, such as the mean change among participants who say their loneliness is "a little better", and it is preferred because it speaks directly to importance. Distribution-based estimates, half a standard deviation of the baseline scores or one standard error of measurement (SEM = SD × √(1 − reliability)), describe a change relative to the spread and the error of the scores, so they serve as checks. To judge a difference in a question, line it up against every MID estimate you are given: in the lesson's example, a fall of 0.4 points is below the anchor-based MID of 1.0 and the distribution-based values of 0.8, so it is not important even if a large trial finds it statistically significant.
  • 5.6 Construct validity evidence is convincing only when the predictions are written down before the results are seen (COSMIN calls this hypothesis testing), so begin any answer by stating what theory expects. In Activity 7.3 the predictions held: the De Jong Gierveld total correlates 0.45 with the UCLA scale and −0.49 with perceived social support (convergent evidence; the sign is negative because more support goes with less loneliness) but only 0.33 with anxiety (discriminant evidence), and the subscale scores fall as the number of close friends rises (known-groups evidence), more steeply for social loneliness (2.47 to 1.13) than for emotional loneliness (2.45 to 1.74), as Weiss's distinction predicts. When you read a correlation matrix, judge the convergent correlations by their absolute size, and remember that a discriminant correlation only needs to be clearly lower than the convergent ones, not zero. Read these correlations with care, too, because they are themselves weakened by unreliability, so the lower reliability of the emotional subscale may explain part of its weaker correlations.
  • 5.7 Whenever you compare mean scores between groups, such as older and younger adults, you are quietly assuming measurement invariance: that the items relate to the construct in the same way in each group. If younger adults read "I miss having people around me" differently from older adults, a difference in means could come from the item's wording rather than from a real difference in loneliness. Invariance is tested by fitting the confirmatory factor model of Section 4 in each group and checking whether the loadings and item intercepts are equal, which goes beyond this lesson. In practice, and in an exam answer, say that the comparison assumes invariance, cite published invariance evidence where it exists, and note any comparison for which it is missing.
Knowledge check: this section

1. A clinic's scale always reads exactly 2 kg heavier than a person's true weight, and two readings of the same person taken a minute apart never differ by more than 0.1 kg. A nurse writes that "the readings are reliable, so they are valid measures of weight." Which statement is correct?

This confuses being wrong with being inconsistent. Readings that differ by at most 0.1 kg have little random error, which is what reliability describes; the constant 2 kg excess is a systematic error, a problem of validity.
An error that is the same for everyone is still systematic error, and validity concerns systematic error. Readings that are consistently 2 kg too high do not reflect true weight, however consistent they are.
Close agreement between repeated readings shows little random error, so the readings are reliable, but the constant excess is systematic error, so they are biased. Reliability is necessary for validity but never sufficient to establish it.
Reliability concerns random error only. A measure can be highly reliable while carrying a systematic error, so these readings are reliable but not valid, rather than neither.
The correct answer is c. The information needed is the kind of error: readings that agree closely with each other show little random error (high reliability), whereas a constant excess of 2 kg is a systematic error, so the readings do not reflect the true weight. Reliability concerns random error and validity concerns systematic error, so a score can be consistent and consistently wrong; reliability is necessary for validity but never enough to establish it. Option a is the tempting error, because it confuses being wrong with being inconsistent. (Section 3, 5.1 Reliability and Validity, Figure 7.4; Objective 5.1)

2. A two-item anxiety screener is given to 300 clinic patients, and in the same week a psychologist who has not seen the screener results carries out a structured diagnostic interview with each patient. Against the interview, a screener score of 3 or more has a sensitivity of 0.80 and a specificity of 0.85. Which statement is correct?

A blinded gold-standard interview at the same time gives concurrent criterion evidence. Sensitivity is the share of people with the disorder whom the cut-point flags, and specificity is the share without it whom the cut-point does not flag.
Predictive criterion evidence compares scores with a criterion measured later. Here the diagnostic interview was carried out in the same week, so the evidence is concurrent.
The design is identified correctly, but sensitivity is turned around. Sensitivity is the share of patients with the disorder whom the screener flags, not the share of flagged patients who have the disorder.
Agreement with a gold standard is criterion evidence, not convergent evidence, which compares the score with another measure of a related construct. The figure for missed cases comes from sensitivity; specificity describes people without the disorder.
The correct answer is a. The information needed is the comparison (a gold standard, the structured interview, measured at the same time, which makes this concurrent criterion validity) and the definitions: sensitivity is the share of people with the condition whom the cut-point flags (0.80, so it misses about 2 in 10), and specificity is the share of people without the condition whom it correctly does not flag (0.85, so about 15 in 100 are wrongly flagged). Option c is the tempting error: it turns the sensitivity around, because the sensitivity is the share of patients with the disorder whom the screener flags, not the share of flagged patients who have the disorder. Option d swaps the meaning of specificity and calls agreement with a gold standard convergent evidence. (Section 3, 5.2 Kinds of Validity Evidence, the Criterion Validity flip card and the paragraph on sensitivity and specificity; Objective 5.2)

3. Before running Activity 7.3, Amira wrote down her predictions: the De Jong Gierveld total should correlate positively with the UCLA Loneliness Scale, negatively with perceived social support and less strongly with anxiety symptoms, and social loneliness should be lower among people with more close friends. The output is summarized below. Which interpretation is best?

Result in the CSCS (3,053 people)Value
Correlation of the total with the UCLA Loneliness Scale0.45
Correlation of the total with perceived social support−0.49
Correlation of the total with anxiety symptoms (GAD-2)0.33
Mean social subscale score (0 to 3): no close friends, 1–2, 3–4, 5 or more2.47, 2.08, 1.78, 1.13
The sign of a convergent correlation follows the theory. More social support should go with less loneliness, so a negative correlation is what Amira predicted and supports the scale.
Each result is matched to the kind of evidence it provides and checked against the prediction written in advance: strong correlations with related measures are convergent, a clearly weaker one with a distinct construct is discriminant, and the gradient across groups is known-groups evidence.
A moderate correlation with a related but distinct construct is expected, and it is clearly weaker than the convergent correlations. That pattern is discriminant evidence that the scale measures loneliness rather than anxiety.
Criterion evidence needs a gold standard, and the lesson notes there is none for loneliness. A difference in the predicted direction between groups that should differ, such as people with more or fewer close friends, is known-groups evidence.
The correct answer is b. The information needed is the prediction written down in advance and the size and sign of each value. A strong correlation with another measure of the same or a closely related construct is convergent evidence, and its sign follows the theory: more support should go with less loneliness, so −0.49 supports the scale, which is why option a is wrong. A clearly weaker correlation with a related but distinct construct (0.33 with anxiety) is discriminant evidence, and a gradient in the predicted direction between groups that should differ (2.47 down to 1.13) is known-groups evidence. There is no gold standard for loneliness, so option d is wrong. (Section 3, 5.6 Construct Validity Evidence from the CSCS, and Activity 7.3; Objectives 5.6 and 5.2)

4. A self-efficacy scale was developed in 2015 with university students, and content, convergent and structural evidence supported its use in research comparing groups of students. A health authority now wants to use its scores to decide which older adults in long-term care are offered a coaching program. A colleague says: "The scale was validated in 2015, so no further evidence is needed." Which response is best?

Validity is not a fixed property of a scale. It is a judgement about how well the evidence supports a particular interpretation and use of scores in a particular population, so evidence from students does not transfer automatically.
Alpha concerns random error, not whether the scores mean the same thing in older adults or support decisions about individuals. Decisions about individuals would in any case need a reliability of about 0.90.
The response is right to disagree but wrong about the evidence. Validity is argued from several kinds of evidence, such as content, convergent and structural evidence, assembled across studies, not from one gold-standard study.
Validity is argued for a stated use in a stated population. The evidence for comparing groups of students supports, but does not settle, the new use of deciding about individual older adults, which needs evidence of its own.
The correct answer is d. The information needed is the original use and population (comparing groups of students) and the new ones (decisions about individual older adults). Validity is a judgement about how well the evidence supports a particular interpretation and use of scores in a particular population, assembled from several kinds of evidence and added to by each study, so the 2015 evidence does not transfer automatically. Option b is the tempting error: an alpha of 0.70 concerns random error, not whether the scores mean the same thing in older adults, and decisions about individuals would in any case need a reliability of about 0.90. Option c is wrong because the argument draws on several kinds of evidence, not on one study. (Section 3, 5.3 Validity as an Argument; Objective 5.3)

5. A team plans to compare the mean loneliness scores of English-speaking respondents, who answered a scale in English, and French-speaking respondents, who answered a French translation. Before comparing the means, what should the team show, and where does the COSMIN taxonomy place that evidence?

Adequate reliability in each language group does not show that the items mean the same thing in both. Responsiveness, in COSMIN, is the ability of scores to detect change over time, which is not the issue here.
Comparing means across language groups assumes that the items relate to loneliness in the same way in each, which is measurement invariance. COSMIN lists this cross-cultural validity under construct validity, beside structural validity and hypothesis testing.
The minimal important difference addresses whether a difference matters, which COSMIN treats under interpretability. It does not show that a translated scale measures the same thing, and criterion validity needs a gold standard.
Stability over occasions is reliability in COSMIN, not structural validity, and it does not show that the two language versions measure loneliness in the same way. That needs evidence of measurement invariance.
The correct answer is b. A comparison of mean scores between groups assumes that the items relate to the construct in the same way in each group; if a translated item is understood differently, the means could differ because of the wording rather than because of loneliness. Measurement invariance is tested by fitting the confirmatory factor model in each group and checking whether the loadings and item intercepts are equal, or it is cited from published evidence, and Table 7.2 lists this cross-cultural validity under construct validity, beside structural validity and hypothesis testing. Option a is tempting, but adequate reliability in each group does not show that the scores mean the same thing in both, and responsiveness is the ability to detect change over time. (Section 3, 5.4 The COSMIN Taxonomy, Table 7.2, and 5.7 Measurement Invariance; Objectives 5.4 and 5.7)

6. A loneliness scale has a reliability of 0.84. In a follow-up survey, participants who said they felt "a little less lonely" had improved by 2.5 points on average. A community program lowers the mean score by 1.2 points compared with a control group (p = 0.02). The analyst prints the two distribution-based checks in R from the baseline scores. Is the change important?

> sd(base$lonely)
[1] 4
> sd(base$lonely) / 2               # half an SD
[1] 2
> sd(base$lonely) * sqrt(1 - 0.84)  # SEM
[1] 1.6
Statistical significance shows whether a difference is likely to be real, not whether it matters. Importance is judged against the minimal important difference, and this change falls below every estimate of it.
Reliability concerns random error, not the size of change that matters. Importance is judged with the minimal important difference, and the program's change is smaller than the anchor-based and distribution-based estimates.
The change is smaller than the anchor-based minimal important difference and below both distribution-based checks, so it is probably not important despite p = 0.02. The anchor-based value is preferred because it reflects what people regard as important.
The minimal important difference does not depend on a p-value. It is a judgement of the size of change that people regard as important, and here it can be applied directly to the observed change.
The correct answer is c. The information needed is the anchor-based estimate (2.5 points, the mean change among people who felt a little better) and the two distribution-based checks that R prints: half a standard deviation (2.0 points) and the standard error of measurement, SD × √(1 − reliability) (1.6 points). A change of 1.2 points is below all three, and the anchor-based value is preferred because it links scores to what people regard as important. Option a confuses statistical significance with importance, and option b confuses reliability, which concerns random error, with the size of change that matters. (Section 3, 5.5 The Minimal Important Difference, Equation 7.8; Objective 5.5)

7. A falls-prevention program uses a published seven-item Fear of Falling Scale with adults aged 70 and over. In its sample the alpha is 0.91, and, as predicted, the score correlates strongly with a question on worry about falling and is higher among people who have fallen. A colleague writes: "Because alpha is 0.91, the scale is valid, so we can compare the mean scores of women and men." Which response identifies the problems with this sentence?

Both leaps are named. Alpha shows consistency, which is necessary but never sufficient for validity, so validity is argued from evidence such as the convergent and known-groups results; a comparison of group means also assumes measurement invariance.
Concern about redundant items starts above about 0.95, so 0.91 does not suggest that items repeat each other. The real problems are treating alpha as proof of validity and assuming measurement invariance between women and men.
Omega is also a reliability coefficient, so reporting it cannot establish validity. A comparison of women and men still needs evidence that the items relate to fear of falling in the same way in both groups.
A reliability above 0.90 is about precision for individual decisions, not about validity. It does not show that the scores mean the same thing in women and men, which a comparison of means assumes.
The correct answer is a. The sentence makes two leaps. First, alpha concerns reliability, the consistency of scores and their freedom from random error; a score can be consistent and consistently measure the wrong thing, so validity is argued from its own evidence, such as the convergent correlation and the known-groups difference here, for a stated use in a stated population. Second, a comparison of mean scores between groups assumes measurement invariance, that the items relate to fear of falling in the same way in women and men; it is tested by fitting the confirmatory factor model in each group, or cited from published evidence and otherwise noted as a limitation. Option b is the tempting error: concern about redundant items starts above about 0.95, and redundancy is not what is wrong with the sentence. Options c and d both treat a reliability coefficient as if it settled validity. (Section 3, 5.1 Reliability and Validity, 5.3 Validity as an Argument and 5.7 Measurement Invariance; Objectives 5.1 and 5.7)
● Complete the quiz to continue.
Section 4 of 4

Latent Variable Models: Factor Analysis and Item Response Theory

⏱ Estimated time: 50 minutes
Lesson 7 · Section 4

Latent Variable Models: Factor Analysis and Item Response Theory

Do the six items measure one construct or two, and how well does each item measure it?

The factor model

Each item is caused partly by a factor

One item in a factor model
\[ \color{#0B7B6B}{\text{item}_i} = \color{#6D28D9}{\lambda_i}\,\color{#065C50}{F} + \color{#C2410C}{u_i} \]
λi loading F latent factor ui unique part of the item
Emotional factorLoadingSocial factorLoading
Emptiness0.82Rely on0.75
Miss people0.20Trust0.73
Rejected0.57Feel close0.65

The two factors correlate 0.25 (standardized estimates from the confirmatory model).

Exploratory and confirmatory

Explore in one half, confirm in the other

Exploratory (EFA)

Every item may load on every factor, and the data suggest the number of factors.

This step uses half 1 (1,708 people).

Confirmatory (CFA)

Each item loads only on its assigned factor, and the model is tested against the data.

This step uses half 2 (1,707 people).

Exploratory factor analysis

Parallel analysis suggests two factors

Parallel analysis plot for the six loneliness items. The first two eigenvalues from the data lie above the eigenvalues from simulated and resampled data, and the remaining four lie below them.
Two data eigenvalues lie above the random-data line.
  • The overall KMO is 0.68, which is adequate.
  • Parallel analysis keeps two factors.
  • Oblimin rotation allows the factors to correlate (r = 0.29).
  • The miss item loads below 0.3 on both factors.
Confirmatory factor analysis

The model is written in lavaan syntax

model_2f <- '
  emotional =~ emptiness + miss + rejected
  social    =~ rely + trust + close
'
cfa_2f <- cfa(model_2f, data = cfa_half)
summary(cfa_2f, fit.measures = TRUE, standardized = TRUE)
  • The operator =~ is read as "is measured by".
  • The first loading of each factor is fixed at 1 to set its scale.
  • The Std.all column holds the standardized loadings.
Model fit

Two factors fit far better than one

IndexGuide for good fitTwo factorsOne factor
CFI0.95 or higher0.9490.739
TLI0.95 or higher0.9050.565
RMSEA0.06 or lower (0.08 acceptable)0.0820.177
SRMR0.08 or lower0.0600.110

The chi-square difference test gives 386.95 on 1 degree of freedom (p < 0.001), and the AIC is 20,825 for two factors against 21,210 for one.

Running case

One construct or two?

  • The two-factor model fits far better than the one-factor model in a separate half of the sample.
  • The fit of the two-factor model is acceptable, with CFI = 0.949 and RMSEA = 0.082.
  • The miss item loads weakly (0.20), which accounts for part of the misfit.
  • The emotional items are all negatively worded and the social items all positively worded, so wording direction may contribute to the separation.
Item response theory

Each item has its own curve

Two-parameter logistic (2PL) item characteristic curve
\[ P(\text{lonely answer} \mid \color{#0B7B6B}{\theta}) = \frac{1}{1 + e^{-\color{#6D28D9}{a}(\color{#0B7B6B}{\theta} - \color{#C2410C}{b})}} \]
θ latent trait a discrimination (slope) b difficulty (location)

The Rasch model fixes a to be equal for all items, the 2PL lets it vary, and the graded response model handles ordered answers.

Curves from the CSCS

Where on the trait do the items measure well?

Item characteristic curves from a two-parameter logistic model for the three social loneliness items scored 0 or 1. The three curves are steep and nearly overlapping, crossing a probability of 0.5 between theta of about minus 0.5 and minus 0.2.
2PL curves for the three social items (0 or 1 scoring).
Test information curve from a graded response model for the three social loneliness items. Information is high between theta of about minus 1 and 2, with two peaks near 4, and falls toward zero below minus 2 and above 3.
Test information from a graded response model.
Carry forward

Three ways to look at the same items

Classical test theory

It gives one reliability for the whole score in one sample.

Factor analysis

It tests whether the items form the domains the theory describes.

Item response theory

It describes each item along the trait and shows where measurement is precise.

The knowledge check and the final assessment follow the reading.

Overview

Sections 2 and 3 produced two hints that the six De Jong Gierveld items do not form a single construct: the six-item alpha (0.59) is lower than the alpha of the social subscale alone (0.75), and the two subscales correlate only 0.21 and relate differently to other measures. A latent variable model can test this directly. This section teaches the common factor model, exploratory factor analysis (EFA), which lets the data suggest a structure, and confirmatory factor analysis (CFA), which tests a structure stated in advance, and it shows how the results decide how the scale is scored and reported in the Measures paragraph of Amira's revised manuscript. It then introduces item response theory (IRT) at a conceptual level, using curves estimated from the loneliness items, and compares how IRT and classical test theory describe precision.

Learning Objectives

  • 6.1 Describe the common factor model, and interpret a standardized loading and its communality.
  • 6.2 Explain why the sample is split at random before a structure is explored and then confirmed.
  • 6.3 Interpret an exploratory factor analysis in R: factorability (the KMO and Bartlett tests), the number of factors from parallel analysis, an oblique rotation and the loading matrix, including weak and cross-loading items.
  • 6.4 Interpret a confirmatory factor analysis fitted in lavaan: judge fit with the chi-square test and the CFI, TLI, RMSEA and SRMR, read the standardized loadings, and compare one-factor and two-factor models.
  • 6.5 Explain what the structural results mean for how a scale is scored, and report the structure and reliability of a scale in a Measures paragraph.
  • 7.1 Explain item difficulty, item discrimination, item characteristic curves and test information, and distinguish the Rasch, two-parameter and graded response models.
  • 7.2 Compare how item response theory and classical test theory describe precision.

The part on latent class analysis near the end of this section is optional enrichment: it introduces a related model and is not one of the lesson objectives.

6.1 The Common Factor Model

A factor model treats the answer to each item as the sum of two parts: a part caused by a latent factor that the items share, and a unique part that belongs to the item alone (Spearman, 1904; Thurstone, 1947). By this we mean that when a person answers "I experience a general sense of emptiness", part of their answer reflects their level of emotional loneliness, which also shapes their answers to the other emotional items, and part reflects things that affect only this item, such as how they interpret the word "emptiness". The loading λ describes how strongly the item depends on the factor. When the items and the factor are standardized, the loading is the correlation between the item and the factor, and its square is the share of the item's variance that the factor explains, called the communality. The rest is the unique variance, which includes measurement error and anything specific to the item's wording. Because the two parts add up to the whole, a high communality and a low unique variance go together, and both describe an item that is a good indicator of the factor. Equation 7.9 writes the model for one standardized item.

The common factor model for one item
\[ \color{#0B7B6B}{\text{item}_i} = \color{#6D28D9}{\lambda_i}\,F + \color{#C2410C}{u_i} \qquad\qquad \text{communality} = \color{#6D28D9}{\lambda_i^2} \qquad \text{unique variance} = 1 - \color{#6D28D9}{\lambda_i^2} \]Eq 7.9
Each standardized item equals its loading times the factor F plus a unique part. The communality is the share of the item's variance that the factor explains, and lavaan prints it in the R-Square section of the summary in Activity 7.4: 0.667 for emptiness, whose standardized loading is 0.82, and only 0.041, about 4%, for miss, whose loading is 0.20.

Squaring is the step most often forgotten: a loading that sounds moderate corresponds to a much smaller share of explained variance, which is why R's R-Square output, rather than the loading itself, is read when the question is how much of an item the factor explains. Figure 7.6 draws the two-factor model of the De Jong Gierveld scale, with the standardized loadings and the factor correlation from the confirmatory model fitted in 6.4.

0.25 Emotionalloneliness Socialloneliness 0.82 0.20 0.57 0.75 0.73 0.65 Emptiness Miss people Rejected Rely on Trust Feel close Standardized loadings and factor correlation from the confirmatory model (1,707 people). Each item also has a unique variance (1 minus its squared loading), not drawn.
Figure 7.6. The two-factor model of the De Jong Gierveld scale. The dashed red path marks the weak item "I miss having people around me".

Factor analysis and the reliability statistics of Section 2 rest on the same reflective model, so they answer related questions. Alpha and omega summarize how well a set of items measures whatever they share, assuming there is one thing they share. Factor analysis asks how many things they share and which items measure each one. A factor model can be used either to find out how many factors a set of items share or to test a structure stated in advance, and the next subsection describes both uses and the order in which Amira applies them.

6.2 Exploratory and Confirmatory Factor Analysis

Exploratory factor analysis lets every item load on every factor and estimates the loadings from the data. It is used when the structure is unknown, for example when a new scale is being developed, and it suggests how many factors there are and which items belong together. Confirmatory factor analysis tests a structure stated in advance: each item loads only on the factor the theory assigns it to, and all other loadings are fixed at zero. A confirmatory model can be rejected by the data, which makes it the stronger test of a published scale's structure.

The two analyses should not be run on the same people. Every sample contains some chance patterns, small correlations that arise from the particular people who happened to take part and would not appear in another sample. An exploratory analysis cannot tell these chance patterns from real structure, so the structure it suggests is partly shaped by them. If the confirmatory model were then tested in the same data, it would fit those same chance patterns again and appear to be confirmed, even if part of what it found would not hold in new data. In other words, exploring and confirming in the same data would let the confirmatory model reward whatever the exploration happened to find. Testing the structure in a separate half of the sample avoids this, because the chance patterns of the first half are not repeated in the second. Amira therefore splits the 3,415 participants at random into two halves of 1,708 and 1,707, explores in the first and confirms in the second. The split uses set.seed(2021) before sample(), so that it is the same every time the code is run and the results can be reproduced, and it matches the split in the Factor Analysis and Scale Scoring walkthrough. The next subsection describes the exploration in the first half.

6.3 Exploratory Factor Analysis in Four Steps

An exploratory factor analysis proceeds in four steps, and the same order is the easiest way to read its output. The first step checks that the items are suitable for factor analysis, with the Kaiser-Meyer-Olkin (KMO) measure and Bartlett's test; in the exploratory half the KMO is 0.68, above the practical minimum of about 0.6. The second chooses the number of factors with parallel analysis, which here suggests two. The third rotates the solution, and because emotional and social loneliness are expected to be related, an oblique rotation is used. The fourth reads the loading matrix: the three social items load on one factor, emptiness and rejection on the other, the item about missing people loads weakly on both, and the two factors correlate 0.29.

The accordion below gives the detail of each step, the guides used to judge it and the values from the exploratory half. Expand each step in turn: Step 2 contains the parallel analysis plot (Figure 7.7), and Step 3 contains a callout that examines the varimax rotation and asks a short question.

Step 1. Check that the items are suitable for factor analysis

Factor analysis needs items that correlate, because a factor is defined by what the items share. The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy checks whether the items share enough. It compares the correlations between items with their partial correlations, which are the correlations that remain between two items once all the other items are held constant. If the items share common factors, much of the correlation between any two items runs through what they share with the rest, so the partial correlations are small compared with the ordinary correlations and the KMO is high. If instead the items correlate only in isolated pairs, the partial correlations stay large and the KMO is low. Kaiser (1974) labelled values of 0.90 or above "marvelous", values in the 0.80s "meritorious", the 0.70s "middling", the 0.60s "mediocre", the 0.50s "miserable" and values below 0.50 "unacceptable", and about 0.6 is often treated as a practical minimum. Bartlett's test of sphericity tests whether the correlation matrix differs from one in which all correlations are zero; with samples of this size it is almost always significant, so it is a minimal check. For the exploratory half, the overall KMO is 0.68 and Bartlett's test gives a chi-square of 1,912 on 15 degrees of freedom (p < 0.001).

Step 2. Choose the number of factors with parallel analysis

Each factor has an eigenvalue that describes how much of the items' variance it explains, with larger eigenvalues for factors that account for more of the shared variation. The old rule of keeping every factor with an eigenvalue above 1 (the Kaiser criterion) often keeps too many factors. The problem is that even random data with no real structure produce some eigenvalues above 1, simply because chance correlations in a finite sample always give the first few factors something to explain. Parallel analysis (Horn, 1965) addresses this by asking how large each eigenvalue would be if there were no structure at all. It generates random data with the same number of people and items, computes their eigenvalues, and compares each eigenvalue from the real data with the corresponding random one. A factor is kept only if its eigenvalue exceeds the random one, because only then does it explain more than chance alone would. For the loneliness items, two eigenvalues lie above the random line, and fa.parallel() reports that "the number of factors = 2".

Parallel analysis plot. The eigenvalues from the data for factors 1 and 2 (about 1.68 and 0.55) lie above the lines for simulated and resampled data; the eigenvalues for factors 3 to 6 lie below them.
Figure 7.7. The blue line shows the eigenvalues from the data and the red lines show those from random data. Two factors lie above the random lines.
Step 3. Rotate the solution

An unrotated solution is hard to read, because most items load on the first factor. The first factor is extracted to explain as much variance as possible, so it tends to pick up a little of every item, and the second factor is left with what remains. Rotation turns the factors so that each item loads strongly on as few factors as possible. By this we mean that rotation does not change how well the model fits the data or how much variance the factors explain together; it only changes how that explained variance is divided between the factors, choosing the division that makes each factor easiest to name. An orthogonal rotation, such as varimax, keeps the factors uncorrelated. An oblique rotation, such as oblimin, allows them to correlate. Constructs in health research are usually related, and emotional and social loneliness are expected to be, so an oblique rotation is the usual choice. If the factors turn out to be uncorrelated, an oblique rotation gives nearly the same answer as an orthogonal one.

A Closer Look: The Varimax Rotation

A two-factor analysis of the depression and anxiety items of the course dataset with a varimax rotation forces the factors to be uncorrelated. Depression and anxiety are expected to correlate, so an oblique rotation such as oblimin is the better choice for those items, as it is for emotional and social loneliness. A factor loading is often defined as the correlation between an item and a factor. That holds only in a one-factor or an orthogonal solution. In an oblique solution, the reported (pattern) loading is the item's weight on one factor with the other factors held constant, and it differs from the item-factor correlation whenever the factors correlate.

Question. What does an oblique solution report that a varimax solution cannot?

AnswerThe correlation between the factors. A varimax solution fixes it at zero, whereas an oblique solution estimates it, as in the correlation of 0.29 between the emotional and social factors in this section. When that correlation is not zero, the oblique solution also keeps the pattern loadings separate from the item-factor correlations.
Step 4. Read the loading matrix

Each row of the loading matrix is an item and each column is a factor. Loadings below about 0.3 are often hidden to make the pattern clearer. A clean structure has each item loading strongly on one factor and weakly on the others. An item that loads weakly on every factor measures little of what the others share, and an item that loads strongly on two factors (a cross-loading) is ambiguous. Factors are named from the content of the items that load on them, because labels such as MR1 and MR2 only number the factors. An item that cross-loads, or loads weakly everywhere, is not dropped automatically: its wording is reviewed for content that touches both domains, and the way it behaves is checked in the confirmatory half of the sample before any decision. In a published scale the item is usually kept and reported, whereas in a scale still under development it may be reworded or removed. In the loneliness data, the three social items load 0.71, 0.74 and 0.68 on the first factor, emptiness and rejection load 0.86 and 0.56 on the second, and the item about missing people loads below 0.3 on both. The two factors correlate 0.29.

Activity 7.4, at the end of 6.4, carries out these four steps in R on the exploratory half of the sample. Its output supplies the values quoted in the accordion, and its call to fa.parallel() draws the plot in Figure 7.7. The exploratory half points to two correlated factors, with the three social items on one, emptiness and rejection on the other, and the item about missing people loading weakly on both. The next subsection tests this structure in the confirmatory half of the sample.

6.4 Confirmatory Factor Analysis in lavaan

A confirmatory analysis has three parts: writing the model, judging how well it fits, and comparing it with a rival model while reading each item's loading.

6.4.1 Writing the Model in lavaan

The lavaan package (Rosseel, 2012) fits confirmatory factor models and structural equation models. A model is written as text, with one line per factor. The operator =~ is read as "is measured by", so social =~ rely + trust + close says that the social factor is measured by those three items. A latent factor has no natural scale. By this we mean that, because the factor is never observed, nothing in the data says whether it should be measured in units like the item codes, in standard deviations, or in any other unit, so the model cannot be estimated until a unit is chosen. To solve this, lavaan fixes the first loading of each factor at 1 by default, which gives the factor the same units as its first item, and the other loadings are then estimated relative to it. This changes how the unstandardized output should be read: the 1.000 in the Estimate column is a choice made to set the scale, not a result, and it is therefore not a sign of a strong item. The standardized loadings (Std.all in the output) are on a common scale and are the values usually reported. With rsquare = TRUE, the summary also prints each item's communality in an R-Square section, and the Std.all value under Variances gives each item's unique variance.

6.4.2 Judging Model Fit

A confirmatory model implies a pattern of correlations among the items. By this we mean that once the loadings and the factor correlation are estimated, the model predicts how strongly every pair of items should correlate. For example, two items that load on the same factor are predicted to correlate according to the product of their loadings, and two items on different factors are predicted to correlate only through the correlation between the factors. Fit statistics measure how closely this implied pattern matches the correlations actually observed in the data. The chi-square test asks whether the misfit could be due to chance. Its weakness is its sensitivity to sample size: with large samples it has the power to detect even tiny differences between the implied and observed correlations, so it rejects almost every model, including models with trivial misfit. For this reason it is reported but rarely decisive. Four indices are usually reported alongside it. The guides for good fit used in this course follow Hu and Bentler (1999), who proposed values close to 0.95 or higher for the CFI and TLI, close to 0.06 or lower for the RMSEA and close to 0.08 or lower for the SRMR. The description of an RMSEA up to 0.08 as acceptable follows Browne and Cudeck (1993), who called such values a reasonable error of approximation. They are rules of thumb, and a model near a guide is better described as borderline than as passing or failing.

Table 7.3 lists the four indices, what each one measures and its guide for good fit, together with the values for the two-factor and one-factor models of the loneliness items that Activity 7.4 produces. Compare each value with its guide before reading the next paragraph.

Table 7.3. Four fit indices for a confirmatory factor model, with the values for the two-factor and one-factor models of the De Jong Gierveld items.

IndexWhat it measuresGuide for good fitTwo-factor modelOne-factor model
Comparative fit index (CFI)Improvement over a model in which no items are related0.95 or higher0.9490.739
Tucker-Lewis index (TLI)The same comparison, with a penalty for model complexity0.95 or higher0.9050.565
Root mean square error of approximation (RMSEA)Misfit per degree of freedom0.06 or lower; up to 0.08 often called acceptable0.082 (90% CI 0.068 to 0.097)0.177
Standardized root mean square residual (SRMR)Average gap between observed and implied correlations0.08 or lower0.0600.110

The two-factor model meets the guide for the SRMR (0.060) but falls short of good fit on the other three indices: the CFI (0.949) is just below 0.95, the TLI (0.905) is below it, and the RMSEA (0.082) is just above even the 0.08 often called acceptable. Its fit is therefore best described as acceptable or borderline, rather than good. The one-factor model fails every guide.

6.4.3 Comparing Models and Reading Each Item

Two models fitted to the same data can be compared directly. When one model is a special case of the other, the two models are said to be nested. For example, the one-factor model is the two-factor model with the factor correlation fixed at 1, because two factors that correlate perfectly are in effect a single factor. For nested models a chi-square difference test asks whether freeing that restriction improves fit by more than chance would, and the AIC and BIC can also be compared, with lower values preferred because they reward fit while penalizing extra parameters. These comparisons are made with anova() in Activity 7.4, where every comparison favours the two-factor model (a chi-square difference of 386.95 on 1 degree of freedom; AIC 20,825 against 21,210).

Good overall fit describes the model as a whole, not each item. Fit indices ask whether the model reproduces the pattern of correlations among the items, and an item that correlates weakly with everything is reproduced well by a small loading, so a model can fit well while one item measures its factor poorly. The standardized loadings (Std.all) are therefore read item by item alongside the fit indices, and the weakest item is the one with the smallest standardized loading. In the confirmatory half, miss has a standardized loading of 0.20, against 0.57 to 0.82 for the other items, and its R-Square is only 0.041.

The card below links again to the narrated walkthrough introduced in Section 2, which also demonstrates both factor analyses on these items and may help before you run the activity.

Learn to do this in R

Narrated R walkthrough: Factor Analysis and Scale Scoring

The Factor Analysis and Scale Scoring walkthrough demonstrates the exploratory and confirmatory analyses on these items line by line with narration, including the parallel analysis plot, the reading of the loading matrix and each part of the lavaan summary.

Open the Factor Analysis and Scale Scoring walkthrough

Activity 7.4 carries out the whole analysis of 6.2 to 6.4 in R: it splits the sample, runs the four steps of the exploratory analysis in the first half, and fits and compares the confirmatory models in the second. Run the script alongside the activity and answer its questions from your own output.

R Activity 7.4: exploratory and confirmatory factor analysis

Text version of the activity: the code of the run-along script, its console output and notes on reading it.

# install.packages(c("psych", "GPArotation", "lavaan"))   # run once, if not yet installed
library(psych); library(GPArotation); library(lavaan)
load(url("https://raw.githubusercontent.com/jorgeandr3s/heal/main/cscs/public_data/CSCS2025_full_cleaned_deidentified_data_and_metadata.RData"))
data <- data[data$SURVEY_collection_year == 2021, ]   # the 2021 wave
item_names <- c("emptiness", "miss", "rejected", "rely", "trust", "close")
items <- data[, paste0("LONELY_dejong_emotional_social_loneliness_scale_", item_names)]
names(items) <- item_names
codes <- c("No" = 1, "More or less" = 2, "Yes" = 3)   # other answers become NA
items[] <- lapply(items, function(x) unname(codes[as.character(x)]))
items[, 4:6] <- 4 - items[, 4:6]   # reverse rely, trust, close: higher = lonelier
items <- na.omit(items)            # the people who answered all six items
nrow(items)
round(cor(items), 2)               # correlations between every pair of items
Console output
Attaching package: 'GPArotation' The following objects are masked from 'package:psych': equamax, varimin This is lavaan 0.7-3 lavaan is FREE software! Please report any bugs. Attaching package: 'lavaan' The following object is masked from 'package:psych': cor2cov [1] 3415 emptiness miss rejected rely trust close emptiness 1.00 0.16 0.48 0.19 0.12 0.21 miss 0.16 1.00 0.11 -0.11 -0.14 -0.08 rejected 0.48 0.11 1.00 0.12 0.11 0.19 rely 0.19 -0.11 0.12 1.00 0.53 0.49 trust 0.12 -0.14 0.11 0.53 1.00 0.49 close 0.21 -0.08 0.19 0.49 0.49 1.00

Reading the output. The messages printed by library() say that GPArotation and lavaan mask a few psych functions; they do not affect this activity. Of the 4,045 people in the 2021 wave, 3,415 answered all six items. In the correlation matrix the three social items correlate 0.49 to 0.53 with each other and emptiness and rejected correlate 0.48, while the correlations between the two clusters are small (0.11 to 0.21). The miss item correlates weakly with every item and slightly negatively with the social items (−0.08 to −0.14).

set.seed(2021)                     # the same random split every time
efa_rows <- sample(nrow(items), size = round(nrow(items) / 2))
efa_half <- items[efa_rows, ]      # half 1: exploratory
cfa_half <- items[-efa_rows, ]     # half 2: confirmatory
c(nrow(efa_half), nrow(cfa_half))
Console output
[1] 1708 1707

set.seed(2021) makes the random split the same every time the code is run. sample() picks 1,708 row numbers at random for the exploratory half, and the minus sign in items[-efa_rows, ] keeps the other 1,707 rows for the confirmatory half. Exploring and confirming in the same people would let the confirmatory model reward whatever the exploration happened to find.

KMO(efa_half)                      # overall MSA above 0.6 is usually adequate
cortest.bartlett(cor(efa_half), n = nrow(efa_half))
fa.parallel(efa_half, fa = "fa")   # how many factors? (it also draws a plot)
efa_2 <- fa(efa_half, nfactors = 2, rotate = "oblimin")
print(efa_2$loadings, cutoff = 0.3)   # loadings below 0.3 are hidden
round(efa_2$communality, 2)        # share of each item's variance explained
round(efa_2$Phi, 2)                # correlation between the two factors
Console output
Kaiser-Meyer-Olkin factor adequacy Call: KMO(r = efa_half) Overall MSA = 0.68 MSA for each item = emptiness miss rejected rely trust close 0.60 0.66 0.60 0.72 0.72 0.74 $chisq [1] 1912.278 $p.value [1] 0 $df [1] 15 Parallel analysis suggests that the number of factors = 2 and the number of components = NA Loadings: MR2 MR1 emptiness 0.859 miss rejected 0.558 rely 0.706 trust 0.738 close 0.682 MR2 MR1 SS loadings 1.568 1.123 Proportion Var 0.261 0.187 Cumulative Var 0.261 0.449 emptiness miss rejected rely trust close 0.74 0.09 0.33 0.51 0.52 0.50 MR2 MR1 MR2 1.00 0.29 MR1 0.29 1.00
Parallel analysis scree plot for the six loneliness items in the exploratory half: the first two eigenvalues from the data (1.67 and 0.55) lie above the eigenvalues from simulated and resampled data, and from the third factor on the data line lies at or below them.

Reading the output. The overall measure of sampling adequacy (KMO) is 0.68, above the usual minimum of about 0.6, and Bartlett's test gives a chi-square of 1,912 on 15 degrees of freedom (a p-value printed as 0 means p < 0.001), so the items are suitable for factor analysis. Parallel analysis keeps the factors whose eigenvalues from the data (blue line) are larger than those from random data of the same size (red lines), which gives two factors; it reports "components = NA" because fa = "fa" asked only about factors. MR1 and MR2 only number the factors (MR stands for minimum residual, the estimation method), so the factors are named from their items: the social items load on MR2 (0.706, 0.738 and 0.682) and emptiness and rejected load on MR1 (0.859 and 0.558). The row for miss is blank because its loadings are below 0.3 on both factors, and its communality of 0.09 means that the factors explain only about 9% of its variance. Together the two factors account for 44.9% of the variance of the six items (Cumulative Var). The oblimin rotation lets the factors correlate, which suits two kinds of loneliness that are expected to be related: they correlate 0.29, whereas a varimax rotation would force this correlation to be zero.

model_2f <- '
  emotional =~ emptiness + miss + rejected
  social    =~ rely + trust + close
'
cfa_2f <- cfa(model_2f, data = cfa_half)
summary(cfa_2f, fit.measures = TRUE, standardized = TRUE, rsquare = TRUE)
fitMeasures(cfa_2f, c("cfi", "tli", "rmsea", "srmr"))   # the four fit indices
Console output
lavaan 0.7-3 ended normally after 28 iterations Estimator ML Optimization method NLMINB Number of model parameters 13 Number of observations 1707 Model Test User Model: Test statistic 100.708 Degrees of freedom 8 P-value (Chi-square) 0.000 Browne's residual (NT model-based) test Test statistic 99.599 Degrees of freedom 8 P-value (Chi-square) 0.000 Model Test Baseline Model: Test statistic 1847.116 Degrees of freedom 15 P-value 0.000 User Model versus Baseline Model: Comparative Fit Index (CFI) 0.949 Tucker-Lewis Index (TLI) 0.905 Loglikelihood and Information Criteria: Loglikelihood user model (H0) -10399.680 Loglikelihood unrestricted model (H1) -10349.325 Akaike (AIC) 20825.359 Bayesian (BIC) 20896.112 Sample-size adjusted Bayesian (SABIC) 20854.812 Root Mean Square Error of Approximation: RMSEA 0.082 90 Percent confidence interval - lower 0.068 90 Percent confidence interval - upper 0.097 P-value H_0: RMSEA <= 0.050 0.000 P-value H_0: RMSEA >= 0.080 0.627 Standardized Root Mean Square Residual: SRMR 0.060 Goodness of Fit Index: Goodness of Fit Index (GFI) 0.982 90 Percent confidence interval - lower 0.976 90 Percent confidence interval - upper 0.988 Parameter Estimates: Standard errors Standard Information Expected Information saturated (h1) model Structured Latent Variables: Estimate Std.Err z-value P(>|z|) Std.lv Std.all emotional =~ emptiness 1.000 0.617 0.816 miss 0.228 0.040 5.757 0.000 0.141 0.202 rejected 0.676 0.089 7.577 0.000 0.417 0.567 social =~ rely 1.000 0.538 0.751 trust 1.014 0.047 21.367 0.000 0.546 0.731 close 0.865 0.041 20.902 0.000 0.465 0.651 Covariances: Estimate Std.Err z-value P(>|z|) Std.lv Std.all emotional ~~ social 0.084 0.012 7.342 0.000 0.254 0.254 Variances: Estimate Std.Err z-value P(>|z|) Std.lv Std.all .emptiness 0.191 0.049 3.865 0.000 0.191 0.333 .miss 0.465 0.016 28.605 0.000 0.465 0.959 .rejected 0.367 0.026 14.302 0.000 0.367 0.678 .rely 0.224 0.014 16.059 0.000 0.224 0.436 .trust 0.259 0.015 17.375 0.000 0.259 0.465 .close 0.295 0.013 22.023 0.000 0.295 0.577 emotional 0.381 0.052 7.292 0.000 1.000 1.000 social 0.290 0.020 14.756 0.000 1.000 1.000 R-Square: Estimate emptiness 0.667 miss 0.041 rejected 0.322 rely 0.564 trust 0.535 close 0.423 cfi tli rmsea srmr 0.949 0.905 0.082 0.060

Reading the summary. The Model Test User Model chi-square is 100.708 on 8 degrees of freedom (p < 0.001); with 1,707 people this test rejects almost any model, so the fit indices are more informative. CFI is 0.949 and TLI is 0.905, RMSEA is 0.082 (90% CI 0.068 to 0.097) and SRMR is 0.060. (Recent versions of lavaan also print Browne's residual test and the goodness of fit index, which this course does not use.) Under Latent Variables, the first loading of each factor is fixed at 1.000 to give the factor a scale, and the Std.all column gives the standardized loadings: 0.816, 0.202 and 0.567 for the emotional items and 0.751, 0.731 and 0.651 for the social items. Under Covariances, the two factors correlate 0.254 (Std.all). Under Variances, the Std.all value for each item is its unique variance, the share of its variance that the factor leaves unexplained (0.959 for miss). rsquare = TRUE adds the R-Square section, the share of each item's variance that its factor explains (the squared standardized loading): only 0.041, about 4%, for miss, against 0.32 to 0.67 for the other items. The last line repeats the four fit indices on one line, ready for a results table.

Files for this activity Data Code Answer key

R Reflect on what you just ran

Answer each question from your own output, as on the final exam: judge each check, name the factors from their items, and say what the results mean for how the scale is scored.

1. The sample was split at random into two halves, and the exploratory factor analysis was run on the first half (1,708 people). (i) Why was the sample split in half before the factor analysis? (ii) Are the items suitable for factor analysis, how many factors does parallel analysis suggest, and how does it decide? (iii) Name the two factors (MR1 and MR2) from the items that load on them. (iv) Which item does not belong clearly to one factor, and what would you do about it?

Model answerExploratory factor analysis suggests a structure from the data, so the structure has to be tested in data that the exploration did not use: explore in one half (1,708 people) and confirm in the other (1,707). The items are suitable for factor analysis: the overall KMO of 0.68 is above the usual minimum of about 0.6, and Bartlett's test is significant (chi-square 1,912 on 15 degrees of freedom, p < 0.001). Parallel analysis suggests two factors; it keeps the factors whose eigenvalues are larger than the eigenvalues from random data of the same size. MR2 is social loneliness, because rely (0.71), trust (0.74) and close (0.68) load on it, and MR1 is emotional loneliness, because emptiness (0.86) and rejected (0.56) load on it. The item miss (“I miss having people around me”) does not belong clearly to either factor: its loadings are below 0.3 on both, and its communality of 0.09 means that the factors explain only about 9% of its variance. The De Jong Gierveld scale is a published scale, so miss is kept to keep scores comparable with other studies, its wording is reviewed, its behaviour is checked in the confirmatory half and it is reported as a limitation; in a new scale under development, an item like this would be reworded or removed.

2. (i) Judge the fit of the two-factor model using the chi-square test and the four fit indices, with the guides taught in the course. (ii) Using the standardized loadings (Std.all) and the R-Square section of the summary, which item is most weakly related to its factor, what share of its variance does the factor explain, and how might this item help to explain the fit?

Model answerThe chi-square is significant (100.71 on 8 degrees of freedom, p < 0.001), but with 1,707 people it rejects almost any model, so it is reported but not decisive. Against the guides, the SRMR (0.060) meets its guide of 0.08 or lower, but the CFI (0.949) and the TLI (0.905) are below 0.95, and the RMSEA (0.082, 90% CI 0.068 to 0.097) is above 0.06 and just above the 0.08 often called acceptable. The fit of the two-factor model is therefore acceptable, or borderline, rather than good: it falls short of the guides for good fit on three of the four indices. The item miss is most weakly related to its factor, with a standardized loading of 0.20 against 0.57 to 0.82 for the other items, and the R-Square section shows that the emotional factor explains only 0.041, about 4%, of its variance (its unique variance, under Variances, is 0.959). An item that relates weakly to its own factor, and slightly negatively to the social items, leaves correlations that the model does not reproduce well, which adds to the misfit.

3. Run your own code. At the end of the script, type and run the four lines below. They fit a one-factor model, in which one factor (loneliness) is measured by all six items, to the confirmatory half and compare it with the two-factor model.
model_1f <- 'loneliness =~ emptiness + miss + rejected + rely + trust + close'
cfa_1f <- cfa(model_1f, data = cfa_half)
fitMeasures(cfa_1f, c("cfi", "tli", "rmsea", "srmr"))
anova(cfa_1f, cfa_2f)
Does the comparison support two factors or one? Use the fit indices, the chi-square difference test and the AIC and BIC. In one sentence, what do these results mean for how the scale should be scored?

Model answerThe one-factor model fits poorly on every index: CFI 0.739 and TLI 0.565 (far below 0.95), RMSEA 0.177 (far above 0.08) and SRMR 0.110 (above 0.08). It fits much worse than the two-factor model: the chi-square difference is 386.95 on 1 degree of freedom, which is large and highly significant (p < 0.001). The AIC and BIC are lower for the two-factor model (20,825 against 21,210; 20,896 against 21,276), so every comparison favours two factors. The scale should therefore be scored and reported as two subscales, emotional and social loneliness, rather than as a single score. One reason for caution is that all three emotional items are worded negatively and all three social items positively, so part of the separation between the factors may reflect the direction of wording (a method effect) as well as the two kinds of loneliness.
Saved.

Your output from Activity 7.4 contains every value quoted in 6.3 and 6.4, and its last question asks what the comparison means for how the scale is scored, the subject of the next subsection.

6.5 What the Structure Means for Scoring and Reporting

Structural results matter because they decide how a scale is scored. When a two-factor model fits well and clearly better than a one-factor model, the scale is scored and reported as two subscales, with or without the total, rather than as a single score. The results are then written up in the Measures paragraph of the manuscript.

6.5.1 One Construct or Two?

The case card below uses the results of the two factor analyses to answer the reviewer's first question, including the caveats that the data raise.

📋 Running case: one construct or two?

The answer to the reviewer's first question is that the items measure two related constructs. In a half of the sample that the exploration did not use, the two-factor model fits far better than a one-factor model (chi-square difference 386.95 on 1 degree of freedom, p < 0.001; AIC 20,825 against 21,210), and the one-factor model fits poorly on every index. The fit of the two-factor model is acceptable, and it falls short of good fit on three of the four indices: the SRMR (0.060) meets its guide, the CFI (0.949) is just below 0.95, and the RMSEA (0.082) and TLI (0.905) fall short of the guides for good fit. Two features of the data explain part of the misfit. The item "I miss having people around me" has a standardized loading of only 0.20, and it correlates slightly negatively with the social items. All three emotional items are worded negatively and all three social items positively, so part of what separates the factors may be the direction of wording (a method effect) as well as the two kinds of loneliness. Method effects of this kind are well documented for scales that mix positively and negatively worded items (Marsh, 1996; DiStefano & Motl, 2006), and they have been reported in confirmatory analyses of the De Jong Gierveld and UCLA loneliness scales (Penning, Liu & Chou, 2014). Amira's reply reports the two-factor result, analyzes the subscales separately in addition to the total, and states both caveats.

The answer to the reviewer is therefore two related constructs, with two caveats stated openly. The reply is completed by a Measures paragraph that reports these results together with the scoring and reliability results of Sections 1 and 2.

6.5.2 The Measures Paragraph

The Measures paragraph of a manuscript tells readers enough about each scale to judge the scores. It names the scale and its source, describes the items and response options with an example, states the scoring rule (including reverse-coding and any cut-point), describes how missing items were handled, reports the reliability observed in the study sample, and summarizes any structural or other validity evidence produced in the study. Claims such as "a validated scale" are replaced by a statement of what was validated, for what use and in which population.

Worked Example 7.2 applies this structure to Amira's manuscript, drawing on the scoring of Section 1, the reliability estimates of Section 2 and the factor analyses of this section. As you read it, notice where each element of the structure appears and how the weak item and the low reliability of the emotional subscale are carried into the analysis plan.

Worked Example 7.2: Amira's revised Measures paragraph

Loneliness. Loneliness was measured with the six-item De Jong Gierveld Loneliness Scale (De Jong Gierveld & Van Tilburg, 2006), which contains three emotional loneliness items (for example, "I experience a general sense of emptiness") and three social loneliness items (for example, "There are plenty of people I can rely on when I have problems"), each answered "yes", "more or less" or "no". Following the published scoring rule, each item was scored 1 for "more or less" or the lonely answer and 0 otherwise, taking account of the positive wording of the social items, to give emotional and social subscale scores (0 to 3) and a total score (0 to 6). Participants with any missing item (630 of 4,045) were excluded, leaving 3,415. To test the structure of the scale, the sample was split at random. In the first half (n = 1,708), parallel analysis and an exploratory factor analysis with oblimin rotation supported two factors. In the second half (n = 1,707), a two-factor confirmatory model fitted acceptably (CFI = 0.949, TLI = 0.905, RMSEA = 0.082, SRMR = 0.060) and considerably better than a one-factor model (CFI = 0.739, RMSEA = 0.177; chi-square difference = 386.95, df = 1, p < 0.001). Internal consistency, computed from the three-point item codes, was adequate for the social subscale (Cronbach's alpha = 0.75; McDonald's omega = 0.75) and low for the emotional subscale (alpha = 0.51; omega = 0.57), in which the item "I miss having people around me" loaded weakly (standardized loading 0.20). We therefore report the two subscales separately as well as the total, interpret associations with emotional loneliness as likely underestimates, and present a sensitivity analysis using the two remaining emotional items.

Worked Example 7.2 completes Amira's reply to the reviewer: it reports that the six items measure two related constructs, that the social subscale has adequate internal consistency and the emotional subscale low internal consistency, and how the analysis takes account of both findings. The callout below states the general form of the paragraph for other scales.

A Measures Paragraph for Any Multi-Item Scale

The same structure serves any multi-item scale. A Measures paragraph names the scale and cites its development paper, gives one example item and the response options, states the scoring rule and any reverse-coding, reports alpha and omega computed in the study sample, and describes how missing items were handled. When the research question depends on the subscales of a scale, or a reviewer could reasonably question its structure, it adds a confirmatory factor analysis with the four fit indices. The Factor Analysis and Scale Scoring walkthrough and the answer key for this lesson contain the code for each step.

Factor analysis has answered the reviewer's question about the structure of the scale. Item response theory, the subject of the next subsection, examines the same items one at a time and shows where along the trait each item measures well.

7.1 Item Response Theory: Item Curves, Difficulty and Discrimination

Classical test theory and factor analysis describe the scale as a whole. Item response theory (IRT) models how each item behaves along the latent trait, usually written θ (theta) and scaled so that 0 is the average and most people fall between −3 and 3 (Embretson & Reise, 2000). Each item is described by an item characteristic curve, the probability of a given answer at each level of the trait. Item response theory is presented here conceptually; the optional Worked code 7.1 at the end of 7.2 shows how the curves are estimated, and it is not assessed through code.

7.1.1 Difficulty, Discrimination and Three Item Models

For an item scored 0 or 1, the two-parameter logistic (2PL) model describes the curve with two numbers. The difficulty b is the trait level at which the probability of the keyed answer (here, the lonely answer) is 0.5. The name comes from educational testing, where a difficult question is one that only students of high ability answer correctly; for a loneliness item, "difficulty" means how lonely a person must be before they are more likely than not to give the lonely answer. An item with a high b is therefore endorsed only by people high on the trait, and if b is below 0, a person of average trait level has a probability above 0.5 of giving the keyed answer. The discrimination a is the steepness of the curve at that point. By this we mean how quickly the probability of the keyed answer changes as the trait increases. An item with a high a separates people just below b from people just above it sharply, because their probabilities of the keyed answer differ a great deal, and an item with a low a separates them weakly, because its curve rises so gradually that people at quite different trait levels answer in similar ways. Item response models differ in which of these features they allow to vary between items. The Rasch model (Rasch, 1960), also called the one-parameter model, gives every item the same discrimination. Because every item then carries the same weight when a person's trait level is estimated, it does not matter which items a person endorsed, only how many, and the total score carries all the information the answers contain about a person's trait level, which is what is meant by calling it a sufficient summary. The two-parameter model (Birnbaum, 1968) lets discrimination vary between items. The graded response model (Samejima, 1969) extends the 2PL to items with ordered categories, such as "No", "More or less" and "Yes", by estimating one discrimination per item and one threshold for each boundary between adjacent categories.

Equation 7.10 gives the 2PL item characteristic curve and the information that an item provides at each level of the trait, a quantity that 7.2 uses to describe precision.

Two-parameter logistic item characteristic curve and item information
\[ P_i(\color{#0B7B6B}{\theta}) = \frac{1}{1 + e^{-\color{#6D28D9}{a_i}(\color{#0B7B6B}{\theta} - \color{#C2410C}{b_i})}} \qquad\qquad I_i(\theta) = \color{#6D28D9}{a_i}^2\,P_i(\theta)\,\big(1 - P_i(\theta)\big) \]Eq 7.10
The probability of the keyed answer depends on the person's trait level θ, the item's discrimination a and its difficulty b. Each item's information is highest at θ = b, where it equals a² / 4. Information adds across items to give the test information, and the standard error of a person's estimated θ is 1 / √(test information).

The explorer in Interactive 7.2 draws the curve for any combination of a and b, together with the item's information. Move the discrimination slider and watch the curve become steeper and the information peak taller and narrower; then move the difficulty slider and watch both slide along the trait.

📊 Interactive 7.2: Explorer: an item characteristic curve and its information

Probability of the keyed answer at θ = 00.50
Maximum item information (at θ = b)0.56

The three social loneliness items, scored 0 or 1, have a between 1.90 and 2.32 and b between −0.45 and −0.19.

−4−2024 00.51 Latent trait (θ)

Probability of the keyed answer   Item information (scaled so that a = 3 fills the plot)

7.1.2 Curves for the Social Loneliness Items

Figures 7.8 to 7.10 come from the CSCS data. Figure 7.8 shows 2PL curves for the three social items scored 0 or 1 by the published rule. All three curves are steep (a between 1.90 and 2.32) and cross 0.5 close together (b between −0.45 and −0.19). The three items therefore behave almost interchangeably: each one separates people just below the middle of the trait from people just above it, and none of them distinguishes well among people who are very connected or very isolated.

Item characteristic curves from a two-parameter logistic model for the social loneliness items rely, trust and close, scored 0 or 1. The curves are steep and nearly overlap; their difficulties are minus 0.31, minus 0.45 and minus 0.19, and their discriminations are 2.04, 1.90 and 2.32.
Figure 7.8. Item characteristic curves (2PL) for the three social items under the published 0 or 1 scoring, estimated with mirt from the 3,415 people with complete answers. Each dot marks the item's difficulty b.

A graded response model keeps all three answer categories. It gives each item a discrimination and two thresholds: b1, the trait level at which a person becomes more likely than not to give "More or less" or the lonely answer, and b2, the level at which the lonely answer itself becomes more likely than not. For the three social items, b1 lies between −0.43 and −0.21 and b2 between 0.99 and 1.31, so all three items measure the same part of the trait. The next subsection shows what these thresholds mean for the precision with which the items measure each person.

7.2 Precision in Item Response Theory and Classical Test Theory

The most practical difference between the two frameworks is how they describe precision. Classical test theory gives one reliability, such as alpha or omega, for everyone in the sample, which implies that the scale measures a very lonely person as precisely as a moderately lonely one. Item response theory instead gives precision that varies along the trait. The reasoning runs in three steps. First, each item's information peaks near its difficulty or thresholds, because an item is most useful for telling apart people whose trait levels sit near the point where its curve rises most steeply; for people far above or far below that point, almost everyone gives the same answer, so the item tells us little. Second, the items' information adds up to the test information curve, which shows how much the whole set of items tells us at each trait level. Third, the standard error of a person's estimated trait is 1 / √(test information), so it is small where the items measure well and large where they do not. In other words, a scale can be precise for some people and imprecise for others, and a single reliability coefficient hides this. For the three social items, the test information is high between a θ of about −1 and 2, with two peaks that correspond to the two sets of thresholds, and it falls toward zero below about −2 and above about 3. Where the information is about 4, the standard error is about 0.5, whereas information near zero gives a very large standard error, so the items distinguish poorly among people who are very connected or very isolated.

The two tabs below show the graded response model results. Open the first tab for the test information curve just described (Figure 7.9), and the second for the category curves of each item (Figure 7.10), whose crossing points are the thresholds that produce the two information peaks.

Test information curve for the three social loneliness items from a graded response model. Information rises from near zero at theta of minus 2 to two peaks of about 4 near theta of minus 0.3 and 1.1, then falls to near zero above theta of 3.
Figure 7.9. Test information for the three social items (graded response model, drawn by plot(grm, type = "info")).
Category response curves for the three social loneliness items from a graded response model. For each item, the curve for category 1 falls, the curve for category 2 rises and falls with a peak near theta of 0.5, and the curve for category 3 rises.
Figure 7.10. Category curves for each social item (drawn by plot(grm, type = "trace")). P1 is the least lonely answer, P2 is "More or less" and P3 is the lonely answer.

A question about how precisely a scale measures at a particular cut-point or trait level is therefore answered from the information or the standard error at that point, not from a single alpha. Item response theory and classical test theory differ in other ways too, and Table 7.4 compares them on six features, from the unit of analysis to the demands that each places on the analyst.

Table 7.4. Classical test theory and item response theory compared.

FeatureClassical test theoryItem response theory
Unit of analysisThe total scoreEach item and its curve
PrecisionOne reliability (alpha or omega) for everyone in the sampleInformation and standard error that vary along the trait
Dependence on the sampleItem statistics and reliability depend on the sample's spreadItem parameters are, in principle, the same in any group in which the model holds
ScoringSum or mean of item codesEstimated trait level (θ), which can be compared across different item sets
Typical usesReporting a scale in a manuscript; short established scalesBuilding and shortening scales, item banks, computer adaptive tests, testing item bias
DemandsSimple to compute and explainLarger samples, model checks (unidimensionality, local independence) and specialist software

The graded response model above was fitted to the three social items only, because IRT models of this kind assume that the items measure one trait (unidimensionality), and the factor analysis has shown that the six items measure two. For a manuscript that uses an established scale such as the De Jong Gierveld scale, classical reliability and a confirmatory factor analysis are usually sufficient. Item response theory becomes important when a scale is being built, shortened or compared across groups, and the PROMIS item banks used in health research are a prominent example of its use (Cella et al., 2010).

Worked code 7.1 shows how the graded response model for the three social items was fitted with the mirt package and how Figures 7.9 and 7.10 were drawn. It is optional and not assessed; open it if you would like to try the model yourself.

R Worked code 7.1: Optional: a graded response model with mirt
Files for this activity Data: the CSCS 2021 wave, loaded from GitHub in the codeno download needed Answer key (R script)includes this optional code at the end

This box is optional and is not assessed. It shows how the item response curves in the reading were estimated, for anyone who would like to try. It continues from the Section 2 activity and uses the social_items data frame created there. It needs the mirt package, which may take a few minutes to install.

# OPTIONAL: install.packages("mirt")   # run once
library(mirt)
# A graded response model for the three social loneliness items
grm <- mirt(social_items, model = 1, itemtype = "graded", verbose = FALSE)
round(coef(grm, IRTpars = TRUE, simplify = TRUE)$items, 2)
Console output
a b1 b2 rely 2.47 -0.30 1.22 trust 2.32 -0.43 0.99 close 2.03 -0.21 1.31

Each row is an item. The column a is the discrimination (2.47, 2.32 and 2.03, all steep). The columns b1 and b2 are the thresholds: the trait levels at which the probability of answering above the first and above the second category reaches 0.5. All three items have b1 near −0.2 to −0.4 and b2 near 1.0 to 1.3, so they measure the same part of the trait.

plot(grm, type = "trace")    # category curves for each item
plot(grm, type = "info")     # test information curve

These two lines produce no console output. They draw the category curves and the test information curve shown in Figures 7.10 and 7.9.

Item response theory has described each social item along a continuous trait. The lesson's objectives end here; the last part of the section introduces a related latent variable model for readers who want to go further.

Additional Context: Latent Class Analysis

This part is background enrichment rather than one of the lesson objectives, and it is not assessed. Factor analysis and item response theory assume that the latent variable is continuous: people differ by degree along a trait such as loneliness. Latent class analysis is the latent variable model for a categorical latent variable. It assumes that the sample is a mixture of a small number of hidden groups, called latent classes, and that each class has its own probability of each answer to each item (Lazarsfeld & Henry, 1968; Collins & Lanza, 2010). Within a class, the answers to different items are assumed to be unrelated (local independence), so the classes account for all of the association between the items. The model estimates the size of each class and, for every person, a set of class-membership probabilities: the probability of belonging to each class given that person's answers. People are usually placed in their most likely class, and the membership probabilities show how certain each placement is.

The number of classes is chosen by fitting models with one, two, three or more classes and comparing them. The BIC balances fit against the number of parameters, and lower values are preferred, although in large samples it often keeps falling as classes are added. Entropy summarizes how clearly people are assigned to classes, on a scale from 0 to 1, with values near 1 indicating well-separated classes. The size of each class and whether the classes make substantive sense also guide the choice. Factor analysis groups items that measure the same trait, whereas latent class analysis groups people who answer in similar ways. The card below links to an optional narrated walkthrough that fits latent class models to CSCS data in R.

Learn to do this in R

Optional enrichment: Latent Class Analysis in R

The narrated walkthrough fits models with one to five classes to seven checkbox items from the Canadian Social Connection Survey on what people missed when they felt lonely, compares the models with the BIC, describes the three classes it keeps and assigns each person to a class. It goes beyond the knowledge checks of this lesson.

Open the Latent Class Analysis in R walkthrough

This section has used exploratory and confirmatory factor analysis to show that the six De Jong Gierveld items measure two related constructs, explained what that result means for how the scale is scored, and drafted the Measures paragraph that completes Amira's reply. It has also introduced item response theory, in which each item has a curve and precision varies along the trait. The key takeaways below draw the section together before its knowledge check, and the final assessment of the lesson brings the four sections together.

Key Takeaways

  • 6.1 Factor analysis rests on the same reflective idea as alpha: the answer to each item is partly caused by a latent factor that the items share and partly unique to the item, and the unique part includes measurement error and anything specific to the item's wording. For a standardized item on a single factor, the loading (λ) is the correlation between the item and the factor, and squaring it gives the communality, the share of the item's variance that the factor explains; the rest, 1 − λ², is the unique variance. Squaring is the step that is most often forgotten: a loading of 0.70 sounds strong, yet the factor explains only about 49% of that item's variance, and for the weak item "I miss having people around me", with a loading of 0.20, it explains only 0.041, about 4%, the value that the R-Square section of the lavaan summary prints in Activity 7.4. In lavaan output you can read the unique variance directly from the Std.all column under Variances (0.959 for miss), so the share of each item's variance that the factor explains can be read from R rather than worked out by hand.
  • 6.2 Exploratory and confirmatory factor analysis do different jobs. Exploratory factor analysis lets every item load on every factor and suggests how many factors there are and which items belong together, whereas confirmatory factor analysis tests a structure stated in advance, with each item loading only on its assigned factor, and because the data can reject it, it is the stronger test of a published scale. Exploratory factor analysis suggests a structure, and confirmatory factor analysis tests it in data the exploration did not use, so the sample is split at random: explore in one half, confirm in the other. The one-line reason an exam answer needs is that confirming in the same data would simply reward whatever the exploration happened to find. In R, calling set.seed() before sample() makes the split the same every time the code is run, so the results can be reproduced.
  • 6.3 An exploratory factor analysis is easiest to read as four steps, and exam questions usually follow the same order. First, check that the items are suitable: an overall KMO of about 0.6 or more (0.68 in the exploratory half) and a significant Bartlett's test, which with large samples is only a minimal check. Second, choose the number of factors with parallel analysis, which keeps the factors whose eigenvalues exceed those from random data of the same size; fa.parallel() prints the answer, and it is more trustworthy than the old rule of keeping every eigenvalue above 1, which often keeps too many factors. Third, use an oblique rotation such as oblimin, because related constructs such as emotional and social loneliness are expected to correlate (0.29 here), and an oblique rotation gives nearly the same answer as an orthogonal one if they turn out not to. Fourth, read the loading matrix: name each factor from the content of the items that load on it (labels such as MR1 and MR2 only number the factors), and flag any item that loads weakly everywhere, as miss does, or loads on two factors, then review its wording and check it in the confirmatory half rather than dropping it automatically.
  • 6.4 To judge a confirmatory model in lavaan, start with the chi-square test but do not stop there: with large samples it rejects almost every model, including models with trivial misfit, so it is reported but rarely decisive. Then compare the four fit indices with their guides: the CFI and TLI should be 0.95 or higher, the RMSEA 0.06 or lower (up to 0.08 is often called acceptable) and the SRMR 0.08 or lower, and a value close to a guide is better called borderline than a pass or a fail, as for the two-factor model here (CFI 0.949, TLI 0.905, RMSEA 0.082, SRMR 0.060). Next, read the Std.all column under Latent Variables item by item, because good overall fit does not mean that every item is good: the weakest item is the one with the smallest standardized loading, and the 1.000 in the Estimate column for the first item of each factor is fixed by lavaan to set the scale, not a sign of a strong item. Finally, compare one-factor and two-factor models with anova(): a large, significant chi-square difference and lower AIC and BIC (lower is better) all favour the two-factor model, as they do here (chi-square difference 386.95 on 1 degree of freedom; AIC 20,825 against 21,210).
  • 6.5 Structural results matter because they decide how the scale is scored. When two factors fit well and clearly better than one, the scale is scored and reported as two subscales, with or without the total, rather than as a single score; for the De Jong Gierveld items, two related factors fit far better than one, so Amira reports the subscales separately as well as the total and notes that the weak item and the direction of wording (a method effect) may explain part of the misfit. A Measures paragraph then brings the whole lesson together, roughly in this order: name the scale and its source, describe the items and response options with an example, state the scoring rule (including reverse-coding and any cut-point), say how missing items were handled, report alpha and omega in the study sample, and summarize the structural and other validity evidence. Finish by stating what the evidence supports, for which use and in which population, instead of calling the scale "validated", and use Worked Example 7.2 as your model.
  • 7.1 Item response theory looks at the items one at a time, describing each with an item characteristic curve: the probability of the keyed answer (here, the lonely answer) at each level of the latent trait θ, which is scaled so that 0 is the average. In the two-parameter model, the difficulty b is the trait level at which that probability is 0.5, so an item with a high b is endorsed only by people high on the trait, and the discrimination a is the steepness of the curve at that point, so an item with a high a separates people just below b from people just above it sharply. Two quick checks help with exam questions: if b is below 0, a person of average trait level (θ = 0) has a probability above 0.5 of giving the keyed answer, and of two items, the one with the larger a has the steeper curve. The Rasch model gives every item the same discrimination, the two-parameter model lets discrimination vary between items, and the graded response model extends the two-parameter model to ordered answers such as "No", "More or less" and "Yes", with one threshold for each boundary between adjacent categories.
  • 7.2 The most practical difference between the two frameworks is how they describe precision. Classical test theory gives one reliability, such as alpha or omega, for everyone in the sample, whereas item response theory gives precision that varies along the trait: each item's information peaks at its difficulty, the items' information adds up to the test information, and the standard error of a person's estimated trait is 1 / √(test information), so it is small where the items measure well and large where they do not. For the three social items, test information is high between a θ of about −1 and 2 (a peak of about 4 gives a standard error of about 0.5) and falls toward zero at the extremes, so the items distinguish poorly among people who are very connected or very isolated. When a question asks how precisely a scale measures at a particular cut-point or trait level, look at the information or the standard error at that point rather than at a single alpha.
Knowledge check: this section

1. In a standardized factor solution, item X has a loading of 0.30 on its factor and item Y has a loading of 0.80 on the same factor. An analyst squares the two loadings in R. Which statement is correct?

> round(c(X = 0.30, Y = 0.80)^2, 2)
   X    Y
0.09 0.64 
This reads the loading itself as a share of variance. In a standardized solution the loading is the correlation between item and factor, and the share of the item's variance that the factor explains is its square, the communality.
Unique variance is 1 minus the communality, the squared loading, not 1 minus the loading. Item X is a weak item: the factor explains only a small share of its variance, as the printed communality shows.
The squared standardized loading is the communality, the share of an item's variance that the factor explains, and the unique variance is what remains. The R line prints these, showing that X is a weak item and Y a strong one.
A positive sign shows only the direction of the relationship. How well an item measures the factor is shown by the size of its loading, and the squared loadings here differ greatly.
The correct answer is c. In a standardized solution the loading is the correlation between the item and the factor, its square is the communality (the share of the item's variance that the factor explains), and the unique variance is 1 minus the communality. The R line prints the communalities: 0.09 for X, so the factor explains about 9% of its variance and its unique variance is 0.91, and 0.64 (64%) for Y. Option a is the tempting error, reading the loading itself as a share of variance, and option b subtracts the loading instead of its square. Item X is a weak item, like "I miss having people around me", whose R-Square in Activity 7.4 is 0.041, about 4%. (Section 4, 6.1 The Common Factor Model, Equation 7.9; Objective 6.1)

2. A team has 1,000 responses to a new ten-item scale and splits them at random into two halves of 500. In the first half the KMO is 0.81, and parallel analysis gives the eigenvalues below. What should the team conclude and do next?

FactorEigenvalue from the dataEigenvalue from random data
12.901.25
21.301.15
31.051.08
40.601.02
Parallel analysis keeps the factors whose eigenvalues from the data exceed those from random data of the same size, which gives two. Testing the structure in the half the exploration did not use avoids rewarding whatever the exploration happened to find.
Keeping every eigenvalue above 1 is the older rule, which often keeps too many factors. Parallel analysis compares each eigenvalue with the random-data value, and the third eigenvalue falls below its random counterpart.
The number of factors is right, but the confirmation is not. A confirmatory model fitted to the same data used for exploration rewards whatever the exploration found, so the structure is tested in the second half.
A KMO in the 0.80s is called meritorious, well above the practical minimum of about 0.6. The items share enough variance for factor analysis.
The correct answer is a. Parallel analysis compares each eigenvalue from the data with the eigenvalue from random data of the same size and keeps the factors whose eigenvalues are larger: 2.90 > 1.25 and 1.30 > 1.15, but 1.05 < 1.08, so two factors. Option b applies the older rule of keeping every eigenvalue above 1, which often keeps too many factors. The structure is then tested in the half that the exploration did not use, because a confirmatory model fitted to the same data would reward whatever the exploration happened to find (option c), and a KMO in the 0.80s is meritorious, well above the practical minimum of about 0.6 (option d). (Section 4, 6.2 Exploratory and Confirmatory Factor Analysis and 6.3 Exploratory Factor Analysis in Four Steps, accordion Steps 1 and 2; Objectives 6.2 and 6.3)

3. A two-factor and a one-factor confirmatory model of a nine-item scale are fitted in the confirmatory half of a sample (900 people). The results are summarized below. Which conclusion is best?

ResultTwo-factor modelOne-factor model
Chi-square (df), p58.4 (26), p < 0.001368.4 (27), p < 0.001
CFI0.980.81
TLI0.970.75
RMSEA0.0370.119
SRMR0.0350.090
AIC21,40021,708
Chi-square difference test310.0 on 1 df, p < 0.001
With 900 people the chi-square test rejects almost any model, including those with trivial misfit, so it is reported but rarely decisive. The fit indices show that the two-factor model fits well.
Simplicity does not outweigh fit. The one-factor model fails every guide, and the difference test and AIC favour two factors, so a single total would misrepresent the structure.
Good overall fit describes the model as a whole. The standardized loadings are still read item by item, because a model can fit well while one item measures its factor poorly.
The two-factor model meets the guides for CFI, TLI, RMSEA and SRMR, the one-factor model fails them, and the difference test and AIC favour two factors. The structure therefore determines that the scale is scored and reported as two subscales.
The correct answer is d. Judge each model against the guides: CFI and TLI of 0.95 or higher, RMSEA of 0.06 or lower and SRMR of 0.08 or lower. The two-factor model meets all four (0.98, 0.97, 0.037 and 0.035) and the one-factor model fails all four, and the chi-square difference test (310.0 on 1 df) and the lower AIC (21,400 against 21,708) both favour two factors. Option a is the tempting error: with 900 people the chi-square test rejects almost any model, so it is reported but rarely decisive. Option c is wrong because good overall fit describes the model as a whole, so the standardized loadings are still read item by item. (Section 4, 6.4.2 Judging Model Fit, Table 7.3, 6.4.3 Comparing Models and Reading Each Item, 6.5 What the Structure Means for Scoring and Reporting, and Activity 7.4; Objectives 6.4 and 6.5)

4. Three items of a hearing difficulty scale, each scored 1 for "yes" and 0 for "no", are described by a two-parameter logistic (2PL) model in which the trait θ (hearing difficulty) has mean 0 and standard deviation 1. Which statement is correct?

ItemDiscrimination aDifficulty b
A: "I have trouble following a conversation in a noisy room"2.4−0.5
B: "I turn the television up louder than other people like"0.70.3
C: "I cannot hear the telephone ring in the next room"1.52.2
This confuses discrimination with difficulty. Difficulty b is the trait level at which the keyed answer has a probability of 0.5, so the hardest item is the one with the highest b; discrimination a is the steepness of the curve.
Item C has the highest difficulty, so it is endorsed mainly by people well above the mean trait level, and item A has the largest discrimination, so its curve is steepest at its difficulty and separates people most sharply there.
This confuses location with steepness. Being located near the average trait level does not make an item sharp; item B has a small discrimination, so its curve is flat and separates people weakly.
This reverses the two models. The Rasch model gives every item the same discrimination, whereas the two-parameter logistic model lets discrimination vary, as these items need.
The correct answer is b. The difficulty b is the trait level at which the probability of the keyed answer is 0.5, so item C (b = 2.2) is endorsed mainly by people more than two standard deviations above the mean; the discrimination a is the steepness of the curve at that point, so item A (a = 2.4) separates people just below θ = −0.5 from people just above it most sharply. Option a confuses discrimination with difficulty, and option c confuses location with steepness: the small a of item B (0.7) gives a flat curve. Option d reverses the models: the Rasch model gives every item the same discrimination, and the two-parameter model lets it vary, as these items need. (Section 4, 7.1.1 Difficulty, Discrimination and Three Item Models, Equation 7.10 and Interactive 7.2; Objective 7.1)

5. A depression scale has an alpha of 0.85. An item response model of the same items gives a standard error of the estimated trait of 0.40 at θ = 0 and 1.20 at θ = 2.5. A colleague says: "Alpha is 0.85, so every participant's score is measured with the same precision." Which response is best?

Alpha is a single reliability for the whole sample, whereas item response theory gives a standard error at each trait level. The larger standard error at the severe end shows that few items measure well there.
The two approaches describe precision differently. Classical test theory gives one reliability for everyone, but the item response model shows standard errors that differ along the trait, so precision is not the same for every participant.
This reverses the meaning of a standard error. A larger standard error means a less precise estimate, so people with severe symptoms are measured less precisely than people near the average.
Standard item response models also assume that the items measure one trait. Their advantage here is that they show how precision varies along the trait, not that they drop the unidimensionality assumption.
The correct answer is a. The information needed is what each statistic describes: alpha is one reliability for everyone in the sample, whereas an item response model gives test information, and a standard error equal to 1 ÷ √(information), at each trait level. A standard error of 0.40 at θ = 0 against 1.20 at θ = 2.5 means that the estimate for a person with severe symptoms has a standard error three times as large, because few items measure well at that end of the trait. Option c reverses the meaning of a standard error, and option d is wrong because standard item response models assume that the items measure one trait. (Section 4, 7.2 Precision in Item Response Theory and Classical Test Theory, Table 7.4; Objective 7.2)

6. A published seven-item Screening Barriers Scale has four practical items (B1 to B4; B4 reads "I do not know how to book a screening test") and three emotional items (B5 to B7). In the confirmatory half of the sample (700 adults), the two-factor model fits well (CFI = 0.983, TLI = 0.972, RMSEA = 0.053, SRMR = 0.031) and far better than a one-factor model (CFI = 0.730, RMSEA = 0.202). Part of the lavaan output for the two-factor model is shown below, followed by one line of R that squares B4's standardized loading. Which statement is correct?

Latent Variables:
                   Estimate  Std.Err  Std.all
  practical =~
    B1                1.000              0.78
    B2                0.962    0.046     0.74
    B3                0.915    0.045     0.71
    B4                0.441    0.044     0.36
  emotional =~
    B5                1.000              0.81
    B6                0.948    0.043     0.77
    B7                0.851    0.042     0.69

> round(0.36^2, 2)
[1] 0.13
Good overall fit means the model reproduces the pattern of correlations, and a weak item is reproduced well by a small loading. The two-factor structure also fits far better than one factor, so a single total would misrepresent it.
lavaan fixes the first loading of each factor at 1.000 to give the factor a scale, so those Estimates are not comparable with the others. The standardized loadings show that B1 and B5 are in fact the strongest items.
In a published scale a weak item is kept, reported and its wording reviewed rather than deleted automatically. Removing it would also not justify a single total, because two factors fit far better than one.
The Std.all column shows B4 with a much lower standardized loading, and its squared loading shows that the factor explains little of its variance. Good fit does not make every item good, and the two-factor structure means two subscales.
The correct answer is d. Read the Std.all column item by item: B4 has a standardized loading of 0.36, against 0.69 to 0.81 for the other items, and the R line shows that its factor explains only 0.13 (about 13%) of its variance. Fit indices ask whether the model reproduces the pattern of correlations among the items, and an item that correlates weakly with the others is reproduced well by a small loading, so a model can fit well while one item measures its factor poorly (option a). In a published scale a weak item is kept, reported and its wording reviewed rather than dropped automatically (option c). Option b is the tempting error: lavaan fixes the first loading of each factor at 1.000 to give the factor a scale, and B1 and B5 have the largest standardized loadings. Because two factors fit well and far better than one, the scale is scored and reported as two subscales. (Section 4, 6.1 The Common Factor Model, 6.4.3 Comparing Models and Reading Each Item, 6.5 What the Structure Means for Scoring and Reporting, and Activity 7.4; Objectives 6.1, 6.4 and 6.5)
● Complete the quiz to continue.
Final Assessment

Lesson 7: Final Assessment

16 questions • 100% required to pass

Bringing It All Together

This lesson took the analyst's view of the multi-item scales that most analyses of the CSCS use. Section 1 traced the chain from a construct, through its domains and items, to an instrument and a scoring rule. It separated reflective scales, whose items are caused by the construct and are expected to correlate, from formative indices, whose components define the construct. It prepared and scored the six De Jong Gierveld items in R, including the reversal of the three positively worded items, and introduced classical test theory, in which reliability is the true-score share of observed variance and measurement error weakens correlations.

Section 2 estimated reliability. A single survey supports only internal consistency, and in the CSCS the social subscale had an alpha and an omega of 0.75 while the emotional subscale had an alpha of 0.51 and an omega of 0.57, with one weak item. Alpha rises with the number of items, assumes equal loadings, cannot show that a scale is unidimensional and has no meaning for an index, and it fell from 0.59 to 0.42 when the reversal step was skipped. Section 3 treated validity as an argument for a stated use in a stated population, organized the evidence with the COSMIN taxonomy, and found convergent, discriminant and known-groups evidence that supports the scale and distinguishes its two subscales. Section 4 used exploratory factor analysis in one half of the sample and confirmatory factor analysis in the other to show that two related factors (CFI 0.949, RMSEA 0.082) fit far better than one (CFI 0.739, RMSEA 0.177), and it introduced item response theory, in which each item has a curve and precision varies along the trait.

Together these steps answered both of the reviewer's questions and produced a Measures paragraph that reports the scale's source, scoring, reliability in the study sample and structural evidence. The final assessment asks for these steps to be applied to new scales.

Key Takeaways from this lesson

  1. You can now trace any score back to its construct, through its domains, items, instrument and scoring rule, and tell a reflective scale from a formative index by asking which way the arrows run. The items of a reflective scale are expected to correlate, whereas the components of a formative index need not, so internal consistency applies only to scales. On the exam, if having one component does not make another more likely, treat the measure as an index and say that alpha does not apply to it.
  2. You can explain how item answers become scores, and check that each step worked. Reverse-worded items must be recoded according to the scoring manual before items are combined. A missed reversal shows up as psych's warning, a negative r.drop and a lower alpha, and the fix is to recode from the manual with (maximum + minimum) minus the original code, not to drop the item or let check.keys = TRUE decide. Remember, too, that a published cut-point gives an easily reported category at the cost of discarding information.
  3. You can say what a reliability value means, what random error does to associations, and which kind of reliability a study design can estimate. A reliability of 0.80 means that 80% of the variance in observed scores is true-score variance, unreliable measures make observed correlations weaker than the true ones, and a single survey supports only internal consistency. When you are given an intraclass correlation, read it with Koo and Li's bands and note that a shift between occasions lowers it; when you are given a kappa, remember that it removes chance agreement and is pulled down by a rare category.
  4. You can read alpha() and omega output and judge it against a stated use. Cronbach's alpha rises with the number of items, assumes equal loadings and cannot show unidimensionality, so McDonald's omega is reported alongside it. As a working rule, about 0.70 suffices in early research, about 0.80 for basic research comparing groups and 0.90 or more for decisions about individuals, and a weak item in a published scale is kept, reported and tested with a sensitivity analysis rather than dropped after looking at the data.
  5. You can explain why a reliable score is not necessarily valid, and build a validity argument from the right kinds of evidence. Validity is an argument for a stated use in a stated population, built from content, criterion, construct and structural evidence. In practice, name the kind of evidence each result provides, compare a difference with the minimal important difference before calling it important, remember that group comparisons assume measurement invariance, and never describe a scale simply as "validated".
  6. You can read exploratory and confirmatory factor analyses and say what they mean for how a scale is scored. Exploratory factor analysis suggests a structure, and confirmatory factor analysis tests it in data the exploration did not use, judged with the CFI, TLI, RMSEA and SRMR. Report the chi-square test but do not let it decide in a large sample, read the standardized loadings item by item, and when two factors fit far better than one, score and report the scale as two subscales.
  7. You can explain item difficulty, item discrimination, item characteristic curves and test information in plain words. Item response theory describes each item with a curve defined by its difficulty and discrimination, and gives precision that varies along the trait. Classical test theory, by contrast, gives one reliability for everyone in the sample, so the most useful reminder is this: difficulty b is where the curve crosses a probability of 0.5, discrimination a is how steep it is there, and the standard error is small only where test information is high.

The final assessment covers all four sections. All 16 questions must be answered correctly (100%) to finish the lesson.

Final Knowledge Assessment

Final assessment: the 16 questions

1. A survey contains the four measures below. For which one is it appropriate to report Cronbach's alpha?

Adverse childhood experiences are separate events that together define exposure; they are not caused by one underlying construct, so they need not correlate. The count is a formative index, and alpha would say nothing about it.
Income, education and employment are conditions that make up area deprivation, so the index is formative. Its components need not correlate, and alpha is not a measure of its quality.
Not smoking, being active and eating enough fruit and vegetables define a healthy lifestyle and need not move together; a person can be active and still smoke. The index is formative, so alpha does not apply.
A person's trust is expected to drive every answer, so the items should correlate and internal consistency is meaningful. This is a reflective scale, the only kind of measure for which alpha is appropriate.
The correct answer is d. The information needed is which way the arrows run for each measure. In the trust scale, a person's trust is expected to drive every answer, so the items should correlate and internal consistency is meaningful (a reflective scale). The other three are formative indices: the experiences, census conditions or behaviours define the construct and need not correlate (a person can be physically active and still smoke), so they would give a low alpha that says nothing about the index. Option a is tempting because it has many yes or no items, but ten separate events are not caused by one underlying construct. (Section 1, 1.2 Scales and Indices: Reflective and Formative Measurement, and the callout "Why the Distinction Matters for Analysis"; Section 2, 1.3 Internal Consistency Applies Only to Scales; Objectives 1.2 and 1.3)

2. A five-item medication worry scale is answered from 1 to 5. Four items are worded so that agreement means more worry; item W3, "I feel confident about my medicines", is worded the other way. Part of the alpha() output for the items as stored is shown below. What does it show?

 raw_alpha
      0.21

 Item statistics
   r.drop
W1   0.36
W2   0.35
W3  -0.51
W4   0.35
W5   0.20
W3 is worded opposite to the other items, and its negative r.drop shows it moves opposite to them, which is the signature of a missed reversal. Recoding it as the manual says and recomputing alpha fixes the scoring.
A missed reversal shows as a negative r.drop, not merely a low one. W5's r.drop is low but positive, which marks a weak item that is scored in the right direction.
This treats a coding error as a bad item. W3's r.drop is not just the lowest but negative, and its opposite wording explains why; the remedy is to recode it, not to drop it.
Differing from zero is not the test; the sign matters. A negative r.drop for an oppositely worded item shows a missed reversal, and it is also pulling alpha down.
The correct answer is a. The information needed is the wording of each item and the sign, as well as the size, of each r.drop, the correlation of the item with the sum of the other items. A negative value means that people who score high on the other items score low on W3, which is what happens when an oppositely worded item is added without reverse-coding, and it also pulls alpha down (to 0.21). W5 has a low but positive value, which marks a weak but correctly coded item, so option b is wrong. Option c treats a coding error as a bad item; the remedy is to recode W3 as 6 minus its code and recompute alpha. (Section 2, 2.3 Spotting a Missed Reversal: The Scoring Manual Decides, and Activity 7.2; Objective 2.3)

3. In a study of social support and wellbeing, the correlation between people's true levels of the two constructs is 0.50. The social support score has a reliability of 0.81 and the wellbeing score has a reliability of 0.49. An analyst applies Spearman's attenuation formula in R. Which interpretation is correct?

> 0.50 * sqrt(0.81 * 0.49)
[1] 0.315
Random error does not average out of a correlation. Spearman's formula shows that the observed correlation equals the true correlation weakened by the reliabilities of the two scores, whatever the sample size.
Spearman's formula multiplies the true correlation by a factor based on the two reliabilities, which the R line computes. Because random error weakens observed associations, the study should expect a correlation well below the true value.
The value R prints is the expected observed correlation, not a reliability. Reliability is defined as the true-score share of observed-score variance in one score, which is a different quantity.
This reverses the direction of attenuation. Random measurement error pulls observed correlations toward zero, so imperfect measures make an association look weaker, not stronger.
The correct answer is b. Spearman's formula (Equation 7.3) gives the observed correlation as the true correlation times the square root of the product of the two reliabilities, which is what the R line computes: it prints 0.315, about 0.32. Option a is the tempting error, assuming that random error has no effect on a correlation, and option d reverses the direction of attenuation. Option c confuses the expected correlation with a reliability, which Equation 7.2 defines as the true-score share of observed-score variance. Because random error weakens observed associations, a correlation estimated with these scores understates the true association. (Section 1, 3.1.2 Measurement Error Weakens Associations, Equation 7.3 and Interactive 7.1; Objective 3.1)

4. An online survey gives a depression scale once to 3,000 adults. In a sub-study, two interviewers independently classify the same 120 participants as having probable depression or not. They agree on 90% of participants. Each interviewer classifies about 1 participant in 10 as "yes", so the agreement expected by chance is 82%, and Cohen's kappa is 0.44. Which statement is correct?

Two interviewers classifying the same people is the design for inter-rater reliability, not test-retest. Raw percentage agreement also includes agreement expected by chance, which is high when one answer is rare, so kappa cannot be set aside.
Inter-rater reliability needs two or more raters assessing the same people. A single self-completed administration, however large, supports only internal consistency.
Two independent raters give inter-rater reliability, and a single administration gives only internal consistency. Kappa removes the agreement expected by chance, which is high when both raters say "no" for most people, so 90% agreement is only modestly better than chance.
Kappa uses all participants, not only those rated "yes" by both. It is lower because it subtracts the agreement expected by chance, which is high when "yes" is rare.
The correct answer is c. The information needed is the design of each study and the two agreement figures. Two raters assessing the same people independently estimates inter-rater reliability, whereas a single administration with no raters supports only internal consistency (test-retest evidence would need a second administration). Kappa removes the agreement expected by chance (Equation 7.5): when each interviewer says "yes" for only 10% of people, most of the agreement comes from both saying "no", so chance agreement is already 82%, and a kappa of 0.44 shows that the 90% agreement is only modestly better than chance. Option a is the tempting error, reading the raw percentage agreement as the reliability. (Section 2, 3.2 Three Kinds of Reliability, Table 7.1 and the callout "What the CSCS Can and Cannot Show", and 3.3.2 Cohen's Kappa for Agreement Between Raters, Equation 7.5; Objectives 3.2 and 3.3)

5. An eight-item scale has an alpha of 0.86, and a parallel analysis of the same items suggests two factors. A colleague says that these two results contradict each other. Which response is best?

This is the common misreading of alpha as evidence of one construct. Alpha rises with the number of items and their average correlation, so items in two correlated clusters can give a high value.
Alpha reflects the number of items and their average correlation, so it can be high when items measure two related constructs. How many constructs underlie the items is answered by factor analysis, so the results do not conflict.
No such rule exists in the lesson. Values above about 0.95 suggest redundant items, not multiple constructs, and alpha cannot show how many constructs a scale measures at any value.
check.keys = TRUE only reverses items flagged by the software; it does not change the number of factors. The two results do not contradict each other in the first place.
The correct answer is b. Alpha depends on the number of items and their average correlation, so eight items in two correlated clusters can reach 0.86 although they measure two constructs; alpha cannot show that a scale is unidimensional. How many constructs underlie the items is answered by factor analysis, here a parallel analysis followed by a confirmatory model. Option a is the tempting error, the common misreading of alpha as evidence of one construct. Option c invents a rule (values above about 0.95 suggest redundant items, not one factor), and check.keys only reverses items. (Section 2, 4.2 What Alpha Assumes and What It Cannot Show, the accordion panel "A high alpha shows that the scale measures one construct" and its callout; Objective 4.2)

6. A six-item work stress scale is answered from 1 (never) to 5 (always), and its alpha is 0.75. Part of the alpha() output is shown below, including the response frequencies for item S6, "I have taken sick leave because of stress". How is S6 best described?

 Reliability if an item is dropped:
   raw_alpha
S6      0.79

 Item statistics
   r.drop
S1   0.59
S2   0.57
S3   0.56
S4   0.54
S5   0.53
S6   0.20

 Non missing response frequency for each item
       1    2    3    4    5
S6  0.68 0.22 0.07 0.03 0.00
An unreversed item has a negative r.drop. S6 has a low but positive r.drop, so it is scored in the right direction but correlates weakly with the other items.
The pile-up is at the lowest answer, "never", which is a floor effect. A ceiling effect is a pile-up at the top answer of the response scale.
A redundant item repeats others and correlates highly with them. S6 has a low r.drop and alpha rises when it is dropped, which marks a weak item, not a redundant one.
Most people answer "never", a floor effect, so S6 separates people only at the high end of stress and correlates weakly with the other items, as its low but positive r.drop shows. In a published scale it would be kept and reported.
The correct answer is d. Read three parts of the output together: S6 has a low but positive r.drop (0.20, against 0.53 to 0.59), alpha would rise from 0.75 to 0.79 without it, and 68% of people answer "never". When most people give the lowest answer the item has a floor effect, so it tells apart only the few people with high stress and relates only loosely to the other items. Option a is the tempting error: an unreversed item has a negative r.drop, not a low positive one, and option b names the wrong end (a ceiling effect is a pile-up at the top answer). In a published scale the item would be kept, reported and tested with a sensitivity analysis rather than dropped. (Section 2, 4.1 Cronbach's Alpha and the Item Statistics, and 4.3 McDonald's Omega and What to Do About a Weak Item, the running case card; Objectives 4.1 and 4.3)

7. A six-item fatigue subscale has an alpha of 0.84 (omega 0.85), and a three-item sleep subscale has an alpha of 0.72 (omega 0.76). A clinic wants to use the subscales both to compare groups of patients in a research report and to decide which individual patients are referred to a fatigue clinic. Which statement is correct?

Against Nunnally's guides, about 0.80 suits research comparing groups and about 0.70 is the lower guide, while 0.90 or more is needed for decisions about individuals. Neither subscale reaches 0.90, so neither should decide referrals alone.
About 0.70 is acceptable in early research and about 0.80 for basic research comparing groups; 0.80 is not a minimum for every use. The sleep subscale is at the lower guide rather than unusable.
This applies a research guide to individual decisions. Using scores to decide which patients are referred needs a reliability of 0.90 or more, which neither subscale reaches.
A slightly higher omega does not change the conclusion. Both omegas remain below the 0.90 or more needed for decisions about individual patients.
The correct answer is a. Compare each value with Nunnally's guides: about 0.70 suffices in the early stages of research, about 0.80 is adequate for basic research comparing groups, and 0.90 or more is needed when scores are used to make decisions about individuals. The fatigue subscale (about 0.85) is adequate for comparing groups and the sleep subscale (0.72 to 0.76) is at the lower guide, but neither reaches 0.90, so neither should be used on its own to refer individual patients. Option c is the tempting error, applying a research guide to decisions about individuals, and a slightly higher omega (option d) does not change the conclusion. (Section 2, 4.1 Cronbach's Alpha and the Item Statistics, and 4.3 McDonald's Omega and What to Do About a Weak Item; Objective 4.1)

8. A report states: "The Food Security Worry Scale is a validated and reliable measure (alpha = 0.86)." Which revision best reflects the lesson's view of validity?

This treats reliability as proof of validity. Alpha concerns random error, whereas validity is a judgement about whether the evidence supports a stated use of scores in a stated population.
Validity is not settled once and for all. Each use and population needs its own argument, and later studies add to the evidence.
The revision keeps reliability separate from validity, names the kinds of evidence, and states the use and population they support, while noting that screening individuals has not been evaluated. That is how the lesson frames a validity claim.
Whether items look relevant to respondents is face validity, the weakest form of evidence. It cannot on its own support a claim that the scale is valid for a stated use.
The correct answer is c. A Measures sentence states what the evidence supports, for which use and in which population, and it keeps reliability (alpha, which concerns random error) separate from validity (whether the scores mean what is claimed for that use). Option a is the tempting error, treating reliability as proof of validity; option b treats validity as a fixed property settled once and for all; and option d relies on face validity, the weakest form of evidence. (Section 3, 5.3 Validity as an Argument; Section 4, 6.5.2 The Measures Paragraph; Objectives 5.3 and 6.5)

9. A pain interference scale is scored 0 to 50. Half its standard deviation is 3.5 points and its standard error of measurement is 3.1 points, and patients who said at follow-up that their pain was "a little better" had improved by 4.0 points on average. Patients at one clinic score 2.0 points lower on average than patients at another clinic (p = 0.01). Which statement is correct?

In a large sample a small difference can be statistically significant without being important. Importance is judged against the minimal important difference, and this difference is below every estimate.
The difference falls below the anchor-based estimate and both distribution-based checks, so it is unlikely to matter to patients. The anchor-based value is preferred because it links scores to patients' own judgement of change.
Distribution-based values describe the spread and error of the scores and say nothing directly about importance. They serve as checks, and the anchor-based estimate is preferred.
The minimal important difference does not depend on a p-value. It describes the smallest difference people regard as important and can be applied to this difference directly.
The correct answer is b. The information needed is the three estimates of the minimal important difference: the anchor-based estimate (4.0 points, the mean change among patients who felt a little better) and the distribution-based checks of half a standard deviation (3.5) and one SEM (3.1). A difference of 2.0 points is below all three. Distribution-based values describe the spread and the error of the scores and say nothing directly about importance, so the anchor-based estimate is preferred and they serve as checks (option c). Option a is the tempting error: in a large sample a small difference can be statistically significant without being important. (Section 3, 5.5 The Minimal Important Difference, Equation 7.8; Objective 5.5)

10. Before analyzing a new caregiver burden score, a team predicted that it would correlate strongly with an established caregiver burden scale and less strongly with depressive symptoms, and that caregivers of people with dementia would score higher than caregivers of people with arthritis. The score correlates 0.68 with the established scale and 0.35 with depressive symptoms, and the mean scores are 34 for dementia caregivers and 26 for arthritis caregivers. Which interpretation is best?

Each result is compared with the prediction made in advance: the strong correlation with an established burden scale is convergent, the clearly weaker one with depression is discriminant, and the predicted group difference is known-groups evidence.
Discriminant evidence needs a correlation clearly lower than the convergent one, not a correlation of zero. Burden and depressive symptoms are related constructs, so a moderate correlation is expected.
A diagnosis in the person receiving care is not a gold-standard measure of the caregiver's burden. A difference in the predicted direction between groups expected to differ is known-groups evidence.
The 0.90 figure is a reliability guide for decisions about individuals, not a validity threshold. A strong correlation in the predicted direction with an established measure is convergent evidence.
The correct answer is a. The information needed is the prediction made in advance and how each result compares with it. A strong correlation with another measure of the same construct is convergent evidence, a clearly weaker correlation with a related but distinct construct is discriminant evidence, and a difference in the predicted direction between groups that should differ is known-groups evidence. Option b is the tempting error: discriminant evidence needs a correlation clearly lower than the convergent ones, and it may still be well above zero, as here. A diagnosis in the person cared for is not a gold-standard measure of the caregiver's burden (option c), and 0.90 is a reliability guide for decisions about individuals, not a validity threshold (option d). (Section 3, 5.2 Kinds of Validity Evidence and 5.6 Construct Validity Evidence from the CSCS; Objectives 5.2 and 5.6)

11. An eight-item barriers-to-healthy-eating scale has four items about time (T1 to T4, for example "I don't have time to cook") and four about cost (B1 to B4, for example "Healthy food costs too much"); item T4 reads "I can't afford the time to plan healthy meals". An exploratory factor analysis with oblimin rotation in the exploratory half of a sample gives the loadings below (loadings under 0.3 are hidden). Which reading is correct?

Loadings:
     MR1    MR2
T1  0.78
T2  0.72
T3  0.69
T4  0.41   0.38
B1         0.80
B2         0.75
B3         0.66
B4         0.58
MR1 and MR2 are placeholder labels from the software. The analyst names each factor from the content of the items that load strongly on it, which here are time items and cost items.
Loading on both factors makes T4 a cross-loading item that does not belong clearly to either, not a strong one. Its wording touches both time and cost, so it is reviewed rather than counted twice.
An item is not dropped automatically, and a structure found in the exploratory half is not confirmed until it is tested in the other half. In a published scale a cross-loading item is usually kept and reported.
Factors are named from the items that load on them, so MR1 is time and MR2 is cost. T4 cross-loads and its wording touches both, so it is reviewed and checked in the confirmatory half before any decision.
The correct answer is d. Factors are named from the content of the items that load strongly on them: T1 to T3 load on MR1 and B1 to B4 on MR2, so MR1 is time barriers and MR2 is cost barriers. T4 loads 0.41 and 0.38, so it is a cross-loading item that does not belong clearly to either factor, and its wording ("afford the time") touches both. Option c is the tempting error: an item is not dropped automatically, and a structure found in the exploratory half is not confirmed until it has been tested in the other half; in a published scale the item is usually kept and reported. (Section 4, 6.3 Exploratory Factor Analysis in Four Steps, accordion Step 4, and Activity 7.4; Objective 6.3)

12. Before an exploratory factor analysis of ten items written to measure two related constructs, worry about health care and avoidance of health care, the output shows an overall MSA (KMO) of 0.86 and a Bartlett test with p < 0.001. The analyst plans an oblimin rotation. Which statement is correct?

Bartlett's test shows only that the correlations are not all zero, which is almost always true in large samples. It says nothing about how many factors the items form.
The KMO is not a share of variance explained. It compares the correlations between items with their partial correlations to show whether the items share enough variance for factor analysis.
A KMO in the 0.80s is meritorious, so the items share enough variance. Because the two constructs are expected to be related, an oblique rotation such as oblimin, which lets the factors correlate, is the usual choice.
This reverses the two kinds of rotation. Varimax is orthogonal and forces the factors to be uncorrelated, whereas an oblique rotation such as oblimin allows them to correlate.
The correct answer is c. The KMO compares the correlations between items with their partial correlations; Kaiser called values in the 0.80s meritorious, and about 0.6 is a practical minimum, so these items share enough variance. Bartlett's test shows only that the correlations are not all zero, which is almost always true in large samples, so it says nothing about the number of factors (option a). Varimax is an orthogonal rotation that forces the factors to be uncorrelated, so option d reverses the two kinds of rotation; for related constructs an oblique rotation is the usual choice, and if the factors turn out to be uncorrelated it gives nearly the same answer. (Section 4, 6.3 Exploratory Factor Analysis in Four Steps, accordion Steps 1 and 3 and the callout in Step 3; Objective 6.3)

13. A two-factor confirmatory model of ten items fitted to 1,500 people gives a chi-square of 402 on 34 degrees of freedom (p < 0.001), CFI = 0.93, TLI = 0.91, RMSEA = 0.085 and SRMR = 0.065. Using the guides taught in the lesson, how is its fit best described?

The lesson's guide for CFI and TLI is 0.95 or higher, not 0.90. Values of 0.93 and 0.91 fall short, so the model is better described as borderline than as fitting well.
Against the lesson's guides, the SRMR meets 0.08 or lower, the CFI and TLI fall below 0.95, and the RMSEA is just above the 0.08 often called acceptable. With 1,500 people a significant chi-square is expected.
In large samples the chi-square test rejects almost any model, including those with trivial misfit, so it is reported but rarely decisive. Fit is judged from the indices read together.
Being close to a guide is not meeting it. The CFI and TLI fall short of 0.95 and the RMSEA exceeds 0.08, so only the SRMR meets its guide.
The correct answer is b. Compare each index with its guide: CFI and TLI of 0.95 or higher (0.93 and 0.91 fall short), RMSEA of 0.06 or lower, with up to 0.08 often called acceptable (0.085 is just above), and SRMR of 0.08 or lower (0.065 meets it). With 1,500 people the chi-square test rejects almost any model, including models with trivial misfit, so it is reported but rarely decisive (option c). Option a uses a looser guide of 0.90 that the lesson does not teach; a model near its guides is better described as borderline than as passing. (Section 4, 6.4.2 Judging Model Fit, Table 7.3; Objective 6.4)

14. A two-factor confirmatory model fits well (CFI = 0.98, RMSEA = 0.04). Part of the lavaan output for the first factor is shown below (=~ is read as "is measured by"), followed by a line of R that squares the standardized loadings. A colleague concludes that "because the model fits well, every item measures its factor well." Which response is best?

Latent Variables:
             Estimate  Std.Err  Std.all
  support =~
    A1          1.000              0.78
    A2          0.950    0.052     0.74
    A3          0.310    0.048     0.26
    A4          0.902    0.050     0.71

> round(c(A1 = 0.78, A2 = 0.74, A3 = 0.26, A4 = 0.71)^2, 2)
  A1   A2   A3   A4
0.61 0.55 0.07 0.50 
The Std.all column shows A3 with a low standardized loading, and its squared loading shows the factor explains little of its variance. Good overall fit describes the model as a whole, and a weak item can be reproduced well by a small loading.
Fit indices ask whether the model reproduces the pattern of correlations. An item that correlates weakly with the others is reproduced well by a small loading, so good fit does not guarantee strong loadings.
lavaan fixes the first loading of each factor at 1.000 to give the factor a scale, so that Estimate is not comparable with the others. A1's standardized loading is actually the highest.
A small standard error shows that an estimate is precise, not that the item is strong. Item strength is read from the standardized loadings, where A3 is the weakest.
The correct answer is a. The standardized loadings in the Std.all column are the values to compare, and the share of an item's variance explained by its factor is the squared loading, which the R line prints: 0.07 (about 7%) for A3, against 0.50 to 0.61 for the other items. Fit indices ask whether the model reproduces the pattern of correlations, and an item that correlates weakly with everything is reproduced well by a small loading, so good fit does not mean that every item is good. Option c is the tempting error: lavaan fixes the first loading of each factor at 1.000 to give the factor a scale, so that Estimate is not comparable with the others, and A1's standardized loading (0.78) is in fact the highest. (Section 4, 6.4.1 Writing the Model in lavaan and 6.4.3 Comparing Models and Reading Each Item; Objective 6.4)

15. A ten-item depression screener has an alpha of 0.85. An item response model of its items shows test information that peaks at about 11 near θ = 1.5, where the screener's cut-point lies, and falls to about 1 at θ = −1. The analyst converts the two information values to standard errors in R. Which statement is correct?

> 1 / sqrt(c(11, 1))
[1] 0.3015113 1.0000000
Alpha is one reliability for the whole sample and hides how precision varies. Item response theory gives a standard error at each trait level, and here it differs greatly between the cut-point and the low end.
This reverses the relationship. The standard error depends inversely on test information, so lower information means a larger standard error and less precision for people with few symptoms.
This reverses the link between discrimination and information. High information near a trait level comes from items with high discrimination and with difficulties near that level.
The standard error is the reciprocal of the square root of test information, as the R line computes, so it is small where information peaks at the cut-point and large where information is low. The screener is precise where its decisions are made.
The correct answer is d. Information adds across items, and the standard error of a person's estimated trait is 1 ÷ √(test information), which is what the R line computes: about 0.30 where the information is 11 and 1.0 where it is 1. A screener whose information peaks at its cut-point is well suited to deciding who lies above or below that point, even though it separates people with few symptoms poorly. Option a is the tempting error: alpha is one reliability for the whole sample and hides how precision varies along the trait. Option c reverses the link, because high information comes from items with high discrimination and with difficulties near that trait level. (Section 4, 7.1.1 Difficulty, Discrimination and Three Item Models, Equation 7.10, and 7.2 Precision in Item Response Theory and Classical Test Theory, Figure 7.9 and Table 7.4; Objectives 7.1 and 7.2)

16. A survey of 2,000 young adults, completed once online, uses the published eight-item Climate Distress Scale, which has a worry subscale and a helplessness subscale. Before the analysis, the team predicted that the total score would correlate strongly with a question on worry about climate change, less strongly with general anxiety, and be higher among people evacuated by a wildfire. The results are summarized below. Which sentence for the Measures paragraph reports them best?

Confirmatory models, random half (n = 1,000)
              CFI   TLI RMSEA  SRMR
Two factors 0.971 0.957 0.055 0.033
One factor  0.867 0.813 0.115 0.064
Chi-square difference test: p < 0.001
Factor correlation: 0.68

Correlations with the total score
  Worry about climate change    0.61
  General anxiety score         0.24

Mean total score
  Evacuated by a wildfire       19.8
  Not evacuated                 15.1
The two-factor model fits well and better than one factor, so the structural evidence supports two subscales, not a single total. Calling the scale "validated" also treats validity as a fixed property rather than stating the use and population supported.
Discriminant evidence needs a correlation clearly weaker than the convergent one, not zero. A higher score after evacuation is known-groups evidence, because evacuation is not a gold standard for distress.
Each result is named as the kind of evidence it provides: structural, convergent, discriminant and known groups. The sentence reports two subscales because two factors fit better, and it ends by stating the use and population the evidence supports.
The chi-square difference test compares the two models, so a small p favours the two-factor model rather than showing that neither fits. The fit indices for the two-factor model meet the guides.
The correct answer is c. Name the kind of evidence that each result provides, and compare the fit with the guides. The two-factor model meets all four guides (CFI and TLI of 0.95 or higher, RMSEA of 0.06 or lower, SRMR of 0.08 or lower) and fits better than one factor, which is structural evidence and means that the scale is reported as two subscales; option a ignores this and also calls the scale "validated". A strong correlation with another measure of climate worry is convergent evidence, a clearly weaker correlation with a related but distinct construct is discriminant evidence even though it is above zero (option b), and a higher score in a group expected to be more distressed is known-groups evidence, not criterion evidence, because evacuation is not a gold standard for distress. Option d misreads the difference test: it compares the two models, and a small p there favours the two-factor model. The best sentence ends by stating the use and population that the evidence supports instead of calling the scale "validated". (Section 3, 5.2 Kinds of Validity Evidence and 5.3 Validity as an Argument; Section 4, 6.4.2 Judging Model Fit, 6.5.2 The Measures Paragraph and Worked Example 7.2; Objectives 5.2, 5.3, 6.4 and 6.5)

✦ Before submitting: pass every section knowledge check (100%).

🏆 Congratulations!

You scored the De Jong Gierveld items, judged their reliability and validity, tested their structure with exploratory and confirmatory factor analysis, and reported the result in a Measures paragraph.

You have successfully completed this lesson: Measurement and Psychometrics.

Your responses have been downloaded automatically.

Next Lesson: Lesson 8 →