HSCI 341 · Lesson 3

Questionnaire
Design

Fundamental Epidemiological Concepts and Approaches

Learning objectives for this lesson:

  1. Distinguish a questionnaire from a survey, and describe the steps in planning a questionnaire: defining objectives, consulting interest holders (for example with the Delphi technique), running focus groups and reviewing existing validated instruments.
  2. Compare in-person, telephone, mailed and internet administration on response rate, cost, interviewer bias, reach and the handling of sensitive questions, and choose a method for a described study.
  3. Describe the four cognitive steps in answering a question (understanding, retrieval, judgement and communication), and identify the step at which a flawed item is likely to fail.
  4. Choose an appropriate question format (open or closed; checklist, multiple-choice, rating or ranking), and explain why exact numbers such as age are preferred to ranges, with income as the usual exception.
  5. Identify the rule of question design that a draft item breaks (keep it short and simple, be specific, avoid double-barrelled and leading wording, and use response categories that are mutually exclusive and jointly exhaustive), and write an improved version.
  6. Describe how to structure and lay out a questionnaire (an introduction, easy opening items, the funnel approach, sensitive items near the end and pre-coded responses), and explain what expert review, field pre-testing, think-aloud interviews, test-retest reliability and validation against a gold standard can each reveal.
  7. Explain why low response threatens validity, recommend strategies to raise response (including follow-up of non-responders), and write a coding plan that gives missing answers, “don’t know” and refusals distinct codes and keeps coding separate from data entry.

This course was developed by Dr. Kiffer G. Card based on Dohoo, I. R., Martin, S. W., & Stryhn, H. (2012). Methods in Epidemiologic Research. VER Inc.

Reference

Glossary: Key Terms, People & Concepts

📚 Reference page, available throughout the lesson

This glossary collects the key concepts, people, and ideas you will meet in this lesson. Use it as a reference while you work through the material, or as a review before assessments. Type in the search box to filter entries.

Key Concepts & Ideas
Questionnaire A structured set of items used to elicit self-reported information from respondents. The instrument operationalises constructs into measurable data.
Item A single question or statement on a questionnaire. Each item targets a specific construct or piece of information.
Construct An abstract concept (e.g., depression, food insecurity) that cannot be directly observed and must be measured indirectly through one or more items.
Scale A composite measure formed from multiple items, typically summed or averaged, intended to capture a single construct with greater reliability than any one item alone.
Likert Scale An ordinal response format (typically 5- or 7-point) ranging from strongly disagree to strongly agree. Strictly speaking the “Likert scale” is the summed score, while the response options are the “Likert items.”
Visual Analog Scale (VAS) A continuous response format on which respondents mark a position along a line (commonly 0–100 mm) representing the magnitude of a feeling or symptom.
Closed-Ended Question An item with a fixed set of response options. Easier to code and analyse but constrains the respondent to predetermined categories.
Open-Ended Question An item that elicits a free-text response. Captures unanticipated answers but requires post-hoc coding and is harder to analyse quantitatively.
Double-Barrelled Question An item that asks about two things at once (e.g., “Do you eat fruits and vegetables?”), making interpretation impossible when respondents disagree on the components.
Leading Question An item phrased to suggest a particular answer, biasing responses toward the implied direction.
Loaded Question An item containing emotionally charged language or controversial assumptions that influence responses regardless of the respondent’s true view.
Social Desirability Bias Tendency for respondents to answer in ways that present them favourably to others, under-reporting socially undesirable behaviours and over-reporting desirable ones.
Response Set / Acquiescence Bias Habitual tendency to answer items the same way regardless of content, for example always agreeing, always selecting the midpoint, or straight-lining a scale.
Recall Bias Differential accuracy of memory by group (often cases vs. controls), which distorts estimates of past exposures.
Reliability The consistency of measurement: whether the instrument yields similar results across raters (inter-rater), occasions (test-retest), or items (internal consistency, e.g., Cronbach’s α).
Validity The degree to which the instrument measures what it claims to measure. Sub-types include face, content, criterion (concurrent/predictive), and construct (convergent/discriminant) validity.
Content Validity The extent to which items adequately cover the breadth of the construct being measured, typically judged by experts.
Construct Validity Evidence that scores on the instrument behave as expected in relation to other measures (convergent and discriminant) and theoretical predictions.
Response Rate The proportion of sampled individuals who complete the questionnaire. Low response rates threaten generalisability through non-response bias.
Methods & Procedures
Cognitive Interviewing A pretest method in which respondents think aloud while answering items, revealing how they interpret questions and arrive at answers. Used to identify wording, recall, and judgement problems.
Pretest / Pilot Study A small-scale administration of a draft questionnaire to identify problems before full deployment: check timing, comprehension, response distributions, and skip-pattern logic.
Filter / Screening Question An item that determines whether subsequent items apply (e.g., “Have you ever smoked?” before smoking-history items). Reduces respondent burden and item irrelevance.
Skip Pattern (Branching Logic) Routing instructions that direct respondents past items not applicable to them, based on prior answers. Critical to test thoroughly during pretest.
Coding Translating raw responses into numeric or categorical values suitable for data entry and analysis. Pre-coding items at the design stage saves time downstream.
Cronbach’s Alpha A measure of internal consistency reliability for multi-item scales. Values of 0.70–0.95 are typically considered acceptable for research use.
No matching entries. Try a different search term.
Section 1

Planning a Questionnaire

⏱ Estimated reading time: 12 minutes

Lesson 3 · HSCI 341

From Who to What and How

After deciding who to sample, you face the next decision: what to ask, and how to ask it.

Section 1 of 4

Planning a Questionnaire

Objectives, focus groups, questionnaire types, and the choice of administration method.

Two words, two meanings

Questionnaire versus survey

Questionnaire

The instrument. A structured set of items designed to elicit specific information from respondents.

Survey

The study. An observational design that collects descriptive information about a population, using questionnaires as its primary tool.

Before you write a question

Four planning steps

1. Define objectives

What variables must be measured to answer your research questions?

2. Consult interest holders

Subject experts, end users of the data, and the target population.

3. Review existing instruments

Search for previously validated questionnaires. Reuse where possible.

4. Engage the population

Members of the group to be surveyed reveal language, concerns, and gaps researchers miss.

A planning tool

Focus groups in questionnaire development

Six to twelve participants, a trained moderator, and a structured discussion.

Focus groups reveal attitudes, surface locally meaningful terminology, clarify study objectives, and identify issues the researcher might never think to ask about. Recording the session for later review is standard practice.

The output is not just ideas for questions. It is a clearer sense of what the construct actually means to the people you are trying to measure.

Two orientations

Qualitative versus quantitative questionnaires

Qualitative (explorative)

Open questions; free responses; used for hypothesis generation and exploration.

Quantitative (structured)

Mostly closed questions; responses coded for statistical analysis; the primary design in epidemiology.

A qualitative phase often comes first, generating the constructs that the quantitative phase then measures.

Administration method

Four methods, each with trade-offs

In-person High response High cost Telephone Mod-high response Moderate cost Mail Low-mod response Low cost Internet Low-mod response Low cost

No method dominates on every dimension. Mixed approaches, such as online surveys with in-person follow-up for non-responders, are common in practice.

Carry forward

What to take into the next section

  • A questionnaire is the instrument; a survey is the study design that uses it.
  • Effective planning starts with objectives and interest holder consultation, not item writing.
  • Administration method shapes response rates, costs, and the potential for bias.
  • Canadian question banks (CCHS, CHMS, and others) offer validated items worth checking before you write new ones.

Overview

The analysis of a survey depends on its sampling design. This lesson takes up the very next decision: once you have decided who to sample, you have to decide what to ask them. The questionnaire is where measurement actually happens, where exposures, outcomes and confounders enter the dataset as raw responses. Done well, it preserves the validity work begun at the sampling stage. Done badly, no amount of clever sampling or statistical modelling later can save the study. The lesson moves from planning a questionnaire, through the choice of question formats and the wording, structure and pre-testing of questions, to response rates and data coding. This first section explains what a questionnaire is and how it differs from a survey, how a questionnaire is planned, the two broad types of questionnaire, and the four ways of administering one.

Learning Objectives

  • 1.1 Define questionnaire and survey, and distinguish between them.
  • 1.2 Explain the key steps in planning a questionnaire, including consultation with interest holders (the Delphi technique), focus groups and the review of existing instruments.
  • 1.3 Compare qualitative and quantitative questionnaire types, and say when each is used.
  • 2.1 Compare the four administration methods on response rate, cost, interviewer bias, reach and the handling of sensitive questions.

1.1 Questionnaires and Surveys

Questionnaires are among the most commonly used tools for collecting data in epidemiological research. The terms questionnaire and survey are often used interchangeably in everyday speech, but in research they have distinct meanings. Click each of the two cards below to read the definition of each term.

QuestionnaireClick to learn more
SurveyClick to learn more

Instrument or Study?

The questionnaire is the instrument; the survey is the study that uses it. Decisions about the survey, such as how the sample is drawn, are separate from decisions about the questionnaire, such as how an item is worded, and both affect how the answers can be interpreted.

The distinction matters because the quality of the instrument limits the quality of the study. As a primary means of data collection, questionnaires play a significant role in the quality of epidemiological research, yet less attention is often paid to the data-collection process than to the analytical methods used later. A well-designed questionnaire is essential for producing reliable, valid data, which Groves & Lyberg (2010) describe as minimising total survey error. Total survey error is the combined amount by which a survey estimate can miss the true population value. By this we mean that error does not come from a single source: it builds up from several, and each one can push the final estimate away from the truth. Sampling error arises because only some members of the population are asked; coverage error arises when the list used to draw the sample leaves some people out; non-response error arises when the people who do not answer differ from those who do; and measurement error arises when the answers people give do not match their true values. Because these errors add together, a survey with a perfect sample can still produce a misleading estimate if the questions themselves are poorly worded. The questionnaire governs measurement error directly, since every confusing or leading item can distort an answer. In addition, through its length and design it also affects non-response, because a long or off-putting form leads more people to give up, a theme taken up in Section 4.

Because the questionnaire carries so much of the weight, it needs its own planning before a single item is written.

1.2 Planning a Questionnaire

Effective questionnaires require careful planning. The very first step is to establish the objectives and information requirements of the study, and this involves consultation with subject experts, with the ultimate users of the information and, ideally, with members of the population to be surveyed. Planning then proceeds through four steps: define the objectives, consult interest holders, review existing instruments and engage the target population. The order matters, because each step shapes what the items must cover and how they should be worded, so item writing comes only after all four. Expand each step below to see what it involves and who takes part.

Step 1: Define Objectives▼

Clearly identify what information the study needs to collect. What are the key research questions? What variables need to be measured? Consulting with subject experts and interest holders at this stage helps ensure all important areas are covered.

Step 2: Consult Interest Holders▼

Engage with subject experts and the end users of the data. If the data will be used by policymakers, they should be part of the planning process, because a questionnaire that omits the information they need cannot inform their decisions. When many interest holders are involved, they will often disagree about what the questionnaire should cover. A structured process like the Delphi technique can help build consensus among these diverse interest holders. By this we mean a method in which a panel of experts answers a series of questionnaires in rounds, usually without meeting face to face. After each round, the organiser summarises the panel's responses and shares the summary, and each member can then revise their views in light of what the others said. Because the rounds are anonymous, no single senior or outspoken person can dominate, and the repeated feedback tends to move the panel toward agreement on which topics and variables matter most.

Step 3: Review Existing Instruments▼

Search for previously published questionnaires on the same topic. These are especially valuable if they have undergone formal validity assessments. By this we mean that other researchers have already tested whether the questions measure what they claim to measure, for example by comparing answers about a condition with clinical records. Because that testing has already been done, building on validated instruments can save time and improve the quality of your data collection, and it also lets your results be compared with those of earlier studies that used the same questions.

Step 4: Engage the Target Population▼

Members of the population to be surveyed should also be consulted during the planning phase. Their input can reveal important issues, concerns, and language preferences that researchers might otherwise overlook.

Two of these steps have well-developed methods of their own: focus groups, which serve the consultation in steps 2 and 4, and the re-use of validated instruments in step 3.

1.2.1 Focus Groups

Focus groups are a valuable tool during questionnaire planning because they consult members of the study population and the end users of the information directly. A focus group consists of 6 to 12 people drawn from the intended study population and/or the end users of the information. An independent moderator guides the discussion to ensure that it stays on track and that no single individual dominates, and it is often helpful to record the discussion for later review. Focus groups provide insight into attitudes, opinions, concerns and experiences. For the questionnaire designer, they help clarify objectives, identify data requirements, define salient terminology and concepts, and surface research issues that need to be addressed. A focus group may show, for instance, that respondents use an everyday word in a different sense from the researchers, in which case the questionnaire must define the term in plain words.

1.2.2 Re-Using Validated Instruments

Step 3 can save more time than any other. Before drafting brand-new items, check whether validated questions already exist in a major Canadian instrument. Re-using items shortens the timeline, because the work of drafting and testing the wording has already been done. In addition, it lets you benchmark your sample against national or provincial norms. By this we mean that when your question is worded exactly as it is in a national survey, your answers can be set directly beside the national figures. For example, if your community survey asks about food insecurity using the same items as the national survey, you can say whether food insecurity in your community is higher or lower than in the province as a whole. If you had written your own question instead, any difference might simply reflect the change in wording rather than a real difference between populations. This immediate population comparison is a major practical advantage.

Four sources are especially useful. Statistics Canada publishes the full Canadian Community Health Survey (CCHS) questionnaire, with derived variables and skip patterns, for every cycle back to 2001; its modules cover chronic conditions, mental health (e.g., PHQ-9, K6), substance use, food security, access to care, and many "rapid response" topical content blocks. The CHMS questionnaire is structured around the direct-measurement clinic visit and is useful for diet recall, physical activity (with accelerometer protocols) and biomarker collection items. The PHAC mental-health and chronic-disease indicator frameworks, such as the Positive Mental Health Surveillance Indicator Framework, specify the question wording behind national surveillance indicators. Finally, BCCDC and provincial community surveys, for example the BC Adolescent Health Survey (McCreary Centre Society), the BC SPEAK community surveys and BCCDC-led behavioural surveillance instruments on STBBI and substance use, supply items for provincial comparisons.

Planning settles what the study needs to know. The next decision is what kind of questionnaire will collect it.

1.3 Qualitative and Quantitative Questionnaires

Questionnaires can be broadly classified as qualitative or quantitative. The two types differ in the kind of questions they use and in the stage of research they serve, and structured, quantitative questionnaires are the main focus of epidemiological research and of the rest of this lesson. Open each tab to compare the two types.

Qualitative (Explorative) Questionnaires

Qualitative questionnaires are sometimes called explorative questionnaires. They consist primarily of open questions that allow participants to express their views and thoughts freely. These are used during the hypothesis-generation phase of research when researchers need to identify as many issues as possible related to the topic. They are often administered through interviews and may be audio or video recorded.

Quantitative (Structured) Questionnaires

Quantitative, or structured, questionnaires are designed to capture specific information about study subjects and their environment using mostly closed questions. The responses can be easily coded and analysed statistically. These are the primary focus of questionnaire design in epidemiological research.

The two types often work in sequence. The focus groups described in 1.2.1 belong to a qualitative phase that finds out which issues matter and how people talk about them, and the quantitative questionnaire that follows measures how common those issues are. To decide between the types for a particular study, look at its aim: exploring what matters points to a qualitative questionnaire, while measuring how common something is or comparing groups points to a quantitative one.

Once the type of questionnaire is settled, the way it is administered affects both the response rate and the quality of the data.

2.1 Methods of Administration

How a questionnaire is administered can have a substantial impact on both the response rate and data quality. There are four methods (in-person interviews, telephone interviews, mailed questionnaires and internet questionnaires), and none is best on every count. Click each card below to see the advantages and disadvantages of one method, then use Table 3.1 to compare all four at once.

In-Person
Interviews
Click to learn more
Telephone
Interviews
Click to learn more
Mailed
Questionnaires
Click to learn more
Internet/Web
Surveys
Click to learn more

Table 3.1 sets the four methods side by side on seven features: response rate, cost, interviewer bias, geographic reach, the use of visual aids, the literacy required of respondents and the handling of sensitive questions. The last row deserves particular attention. When people answer a question aloud to another person, many feel some pressure to give the answer that seems acceptable or expected rather than the one that is true. Answers given to an interviewer may therefore be shaded toward what the interviewer seems to expect. For example, a respondent asked face to face how many alcoholic drinks they have in a week may report fewer than they actually drink. Because no one is listening or watching, questionnaires completed in private by mail or online remove much of this pressure, and so they tend to produce more candid answers to sensitive questions.

Table 3.1. Comparison of in-person, telephone, mail and internet administration of questionnaires across seven features.

FeatureIn-PersonTelephoneMailInternet
Response RateHighModerate–HighLow–ModerateLow–Moderate
CostHighModerateLowLow
Interviewer BiasPossibleReducedNoneNone
Geographic ReachLimitedModerateWideWide
Visual AidsYesNoPrinted images onlyYes
Literacy RequiredNoNoYesYes
Sensitive QuestionsAnswers may be shaded toward what the interviewer seems to expectSome under-reporting of sensitive behavioursSelf-completed in private; answers tend to be more candidSelf-completed in private; answers tend to be more candid

No method wins on every row, so choosing one for a particular study means working through the rows of Table 3.1 in turn. Ask who the respondents are and how they can be reached, whether they can read the form, how sensitive the questions are, and what the budget allows. A survey of parents across a region, for example, might use a mailed or web questionnaire to keep costs low and let parents answer questions about illness in private, with telephone follow-up for those who do not respond. Mixed designs of this kind are common because they combine the strengths of more than one method.

This section has defined questionnaires and surveys, set out the steps of planning, and compared the types of questionnaire and the methods of administering them. The key takeaways below summarise these points, and the next section turns to the choice between open and closed question formats.

Key Takeaways

  • 1.1 People often use the words questionnaire and survey as if they meant the same thing, but in epidemiology they name two different things. A questionnaire is the data-collection tool: the structured set of questions you put in front of respondents, whether in person, by telephone, by mail or online. A survey is the observational study that uses a questionnaire to collect descriptive information about a population, such as how common a disease or an exposure is. The simplest way to keep them apart is to remember that the questionnaire is the instrument and the survey is the study design. When a question asks you to distinguish them, give each its role in a sentence, and remember that survey-level decisions, such as how the sample was drawn, affect how you can interpret the answers the questionnaire collects.
  • 1.2 Good questionnaires are planned long before the first question is written. Start by defining the study objectives: what you actually need to know, and which variables must be measured to answer your research questions. Then consult the people who matter, including subject experts, the end users of the data (such as policymakers) and members of the population you will survey; the Delphi technique is a structured way to build consensus among these interest holders. Focus groups of 6 to 12 people from the study population, guided by an independent moderator, reveal the attitudes, concerns and everyday terms that respondents use. Before drafting new items, search for existing validated instruments, such as the Canadian Community Health Survey, because reusing validated questions saves time and lets you compare your sample with national or provincial norms. If you are asked how to plan a questionnaire, give these steps in order, say what each one adds, and avoid the common trap of starting with item writing.
  • 1.3 Questionnaires come in two broad types, and the stage of your research decides which one you need. A qualitative (explorative) questionnaire uses mainly open questions that let people express their views freely; it suits the hypothesis-generation phase, when you are still finding out which issues matter, and it is often administered through interviews that may be recorded. A quantitative (structured) questionnaire uses mostly closed questions to capture specific variables whose answers can be coded and analysed statistically, and it is the main design in epidemiological research. In practice a qualitative phase often comes first and identifies the ideas that the quantitative questionnaire then measures. When you are asked to choose between them, look at the aim of the study: exploring what matters points to a qualitative questionnaire, while measuring how common something is or comparing groups points to a quantitative one.
  • 2.1 No method of administration is best on every count, so choosing one is always a matter of weighing trade-offs. In-person interviews usually achieve the highest response, allow visual aids and suit respondents with low literacy, but they are expensive, geographically limited and open to interviewer bias, and answers to sensitive questions may be shaded toward what the interviewer seems to expect. Telephone interviews keep many of these advantages at moderate cost with less interviewer bias, but respondents will give only limited time and some under-report sensitive behaviours. Mailed and internet questionnaires are cheap, reach wide areas and have no interviewer bias, and because people complete them in private their answers to sensitive questions tend to be more candid; however, response is lower, respondents must be able to read, and web surveys reach only people with internet access. When a question describes a study, work through the rows of Table 3.1 (who the respondents are and how they can be reached, whether they can read the form, how sensitive the topic is and what the budget allows) and justify your choice on the features that matter most for that study. Mixed approaches, such as a mailed or web questionnaire with telephone follow-up for non-responders, are common and often make the strongest answer.
Knowledge Check: this section

1. A health authority plans to estimate how many adults in the region smoke. It will select 1,200 adults at random and ask each of them to complete a 20-item form about tobacco use. A student writes in the project plan: “Our survey is the 20-item form; the questionnaire is the plan to sample 1,200 adults and estimate how common smoking is.” What is wrong with the student’s statement?

The words are often swapped in everyday speech, but in research they have distinct meanings: the questionnaire is the instrument put in front of respondents and the survey is the study that uses it, so the order matters.
A questionnaire is the data-collection instrument and a survey is the observational study that uses questionnaires to describe a population, so the 20-item form and the sampling plan carry the opposite labels to the ones the student gave them.
The lesson does not restrict surveys to in-person administration: a survey can collect its questionnaires in person, by telephone, by mail or online. What makes this a survey is that it samples a population to describe how common smoking is.
Whether items are open or closed does not decide what counts as a questionnaire. Structured questionnaires are made mostly of closed questions and are still questionnaires; the term names the instrument, whatever its item formats.
Correct answer: the labels are reversed. The information needed is what each part of the plan does: the 20-item form is the tool put in front of respondents, and the plan to sample 1,200 adults and estimate how many smoke is the study. A questionnaire is the data-collection instrument, and a survey is the observational study that uses questionnaires to collect descriptive information about a population, so the student has swapped the two terms. The first option is tempting because the words are often used interchangeably in everyday speech, but in research they have distinct meanings. (Section 1, slide “Questionnaire versus survey” and 1.1 Questionnaires and Surveys; Objective 1.1)

2. A city asks your team for a questionnaire on food insecurity among university students. At the first meeting, a team member proposes to spend the afternoon writing 40 new items. According to the lesson, what should come before any new item is written?

This reverses the planning order. Consultation is meant to shape what the questionnaire measures, so interest holders should help define the objectives before items exist, not react to 40 items written without agreed aims.
Re-using a validated instrument is one of the planning steps, but it comes after the objectives are defined. Without knowing which variables the study must measure, the team cannot judge whether an instrument found online fits.
The administration method matters, but it is not the first planning step. The lesson starts with objectives, consultation, a search for existing instruments and engagement of the target population; the method is then chosen to suit the study.
These are the lesson’s four planning steps: define the objectives and variables, consult interest holders (the Delphi technique builds consensus), review existing validated instruments and engage the target population, for example through a focus group.
Correct answer: define the objectives, consult interest holders, review existing instruments and engage the students. The information needed is where the team is in the process: no objectives have been agreed and no one has been consulted. The lesson’s four planning steps are to define the objectives (what variables must be measured), consult interest holders (the Delphi technique builds consensus among them), review existing validated instruments (subsection 1.2.2 notes that the CCHS has food-security items) and engage the target population, for example in a moderated focus group of 6 to 12 students. Writing the items first and consulting afterwards is tempting because it feels productive, but planning starts with objectives and consultation, not item writing; re-using an existing instrument is part of planning, but only once the objectives say what it must measure. (Section 1, slide “Four planning steps”, 1.2 Planning a Questionnaire, 1.2.1 Focus Groups and 1.2.2 Re-Using Validated Instruments; Objective 1.2)

3. Little is known about why young adults in a rural community avoid the local sexual-health clinic. Before the team measures how common each barrier is, it wants to find out what the barriers are. Which type of questionnaire suits this first phase?

Closed questions are easier to code, but ease of coding is a reason for the later measurement phase. Here the barriers are still unknown, and closed answers can only offer barriers the team has already thought of.
A list chosen in advance assumes the team already knows the barriers, which is exactly what this phase is meant to find out. Respondents could not raise a barrier that is missing from the five listed.
When the aim is to find out what the issues are, the lesson calls for a qualitative, explorative questionnaire of open questions, often given as an interview, which lets respondents name barriers in their own words and generates hypotheses for later measurement.
Borrowing barriers from other communities still fixes the answers in advance and asks only whether they apply. It measures known barriers rather than discovering local ones, so it belongs to a later, structured phase.
Correct answer: a qualitative (explorative) questionnaire. The information needed is the aim of this phase: the team does not yet know what the barriers are, so it is generating hypotheses rather than measuring known variables. Qualitative questionnaires use open questions that let respondents raise issues in their own words, which suits the hypothesis-generation phase; a quantitative, structured questionnaire comes afterwards to measure how common each barrier is. Closed questions are tempting because they are easier to code, but every structured option here can offer only the barriers the team has already thought of, which is exactly what this phase cannot assume. (Section 1, slide “Qualitative versus quantitative questionnaires” and 1.3 Qualitative and Quantitative Questionnaires; Objective 1.3)

4. A provincial programme wants to ask 5,000 adults across the province about alcohol use and sexual behaviour. The budget is small, and nearly all of these adults read well and have internet access. Using Table 3.1, which administration method fits best?

Table 3.1 supports this choice: an internet questionnaire is low in cost and wide in reach, its literacy and access needs are met here, it has no interviewer, and private answers to sensitive questions tend to be more candid. Follow-up offsets its lower response.
In-person interviews do give the highest response, but they are the most expensive method and limited in reach, which rules them out for 5,000 adults on a small budget, and an interviewer may shade sensitive answers.
Telephone interviews reduce interviewer bias rather than remove it, and the lesson notes that they can lead to under-reporting of sensitive behaviours, so both claims in this option run against Table 3.1.
This reverses the effect of an interviewer on sensitive topics. Table 3.1 notes that answers given to an interviewer may be shaded toward what the interviewer seems to expect, and in-person interviews are also the most costly method.
Correct answer: an internet questionnaire, with follow-up of non-responders. The information needed is the size and spread of the sample, the budget, the respondents’ literacy and internet access, and the sensitivity of the topics. Working down Table 3.1, internet administration is low in cost and wide in reach, its literacy and access requirements are met here, it has no interviewer bias, and answers given in private tend to be more candid; in-person interviews are the most expensive, limited in reach and may shade sensitive answers toward what the interviewer seems to expect. In-person interviews are tempting because they give the highest response, but that advantage does not outweigh their cost and reach here, and the lower response of a web survey can be raised by following up non-responders; telephone interviews reduce interviewer bias rather than remove it and tend to under-report sensitive behaviours. (Section 1, slide “Four methods, each with trade-offs”, 2.1 Methods of Administration, Table 3.1 and the paragraph after it; Objective 2.1)

5. A regional health authority will estimate how many adults aged 65 and older had a fall in the past 12 months, and it will repeat the estimate every two years. Early focus groups show that many older adults do not count a slip or a stumble that causes no injury as a “fall”. What should the main questionnaire be, and how should it ask about falls?

The focus groups were the qualitative phase, and they have done their job by revealing the problem with the word “fall”. Measuring and repeating an estimate calls for a structured questionnaire; the finding is fixed by better wording, not by abandoning measurement.
A structured questionnaire is right, but keeping the word “fall” ignores the focus-group finding. Respondents who do not count a slip without injury would leave those events out, so falls would be under-counted.
Restricting the item to injurious falls changes what the study measures. The aim is to estimate how many older adults fell, and the focus groups showed that non-injurious falls are the ones respondents miss.
A structured questionnaire suits an estimate that will be repeated and compared over time, and describing the event in plain words applies the focus-group finding so that slips and trips without injury are counted.
Correct answer: a quantitative questionnaire that defines the event in plain words. The information needed is the aim (to measure how common falls are and to repeat the measure in two years) and the focus-group finding (respondents do not count falls without injury). Measuring and comparing over time calls for a structured questionnaire of mostly closed or numerical items that can be coded and analysed; the focus groups were the qualitative, exploratory phase that comes first, and their finding means the item cannot rely on the word “fall” but must describe the event. A qualitative questionnaire is tempting because people understand the word differently, but that problem was found by the qualitative phase and is solved by better wording, not by giving up measurement; keeping the word “fall” would under-count falls without injury, and asking only about injurious falls changes what the study measures. (Section 1, 1.3 Qualitative and Quantitative Questionnaires and 1.2.1 Focus Groups; Objective 1.3)

6. The same health authority has $9,000 for data collection and needs about 300 completed questionnaires. About 4 in 10 older adults in the region do not use the internet, many have poor eyesight or limited reading ability, and most live in small, widely spread communities. A pilot gave the cost per completed questionnaire as: in-person interview $40, telephone interview $22, mailed questionnaire $8, web survey $5. Which choice is best supported?

The arithmetic is right and the web survey is cheapest, but cost is only one row of Table 3.1. About 4 in 10 of these older adults do not use the internet, so a web survey would miss a large part of the population.
300 × $40 = $12,000 exceeds the $9,000 budget, so this option fails before its advantages count. In-person interviews do give high response and allow visual aids, but the study cannot afford them.
Telephone interviews cost 300 × $22 = $6,600, within the $9,000 budget, need no reading or eyesight, and reach small, scattered communities. The lesson’s trade-off is that respondents give only limited time on the telephone, so the questionnaire must be short.
A mailed questionnaire is cheap, but reaching a household is not the same as obtaining a usable answer: a printed form needs good eyesight and reading ability, which many of these respondents lack, and mailed questionnaires have low response.
Correct answer: telephone interviews. The information needed is the cost of 300 completed questionnaires by each method (web $1,500, mail $2,400, telephone $6,600, in-person $12,000), the $9,000 budget, and the respondents’ internet access, eyesight and reading. In-person interviews exceed the budget. The web survey is cheapest but would miss the 4 in 10 older adults who do not use the internet. A mailed form needs good eyesight and reading, and its response is low. Telephone interviews fit the budget, need no reading and reach small communities, at the cost of the limited time people give on the telephone. The web survey is tempting because it is cheapest, but cost is only one row of Table 3.1, and reach and literacy matter more for this population. (Section 1, 2.1 Methods of Administration and Table 3.1; Objective 2.1)

✦ Pass the knowledge check with 100% to continue

Section 2

Designing Questions: Open & Closed Formats

⏱ Estimated reading time: 15 minutes

Section 2 of 4

Designing Questions: Open & Closed Formats

The cognitive process of answering questions, and the design rules for open and closed formats.

The cognitive model

Four steps in answering a question

1 Understanding

Does the respondent read the question the same way you wrote it?

2 Retrieval

Can they access the information needed to answer?

3 Judgement

Can they map their internal state onto your response categories?

4 Communication

Do the response options faithfully record what they mean?

Schwarz (1999); Tourangeau & Yan (2007).

Open questions

Capture continuous data wherever possible

Prefer exact numbers

"How old are you? ___" captures more information than asking for an age range. Continuous data can always be binned later.

Exception: sensitive quantities

For income or other sensitive data, ranges reduce refusal bias. An acceptable compromise when exact values would deter response.

Always specify units and offer measurement-scale options when numerical responses vary (for example, kilograms or pounds).

Closed questions

Four types, four sets of rules

Checklist

Select all that apply. Options need not be mutually exclusive. Each becomes a separate binary variable in the dataset.

Multiple-choice

Select one. Categories must be mutually exclusive and jointly exhaustive. Add "Other" to ensure the latter.

Rating (Likert)

Minimum 5 points; include a midpoint or force choice; always add a "Don't know" option.

Ranking

Order by priority. Cognitively demanding for long lists; inter-rank intervals are unknown.

The Likert scale

Design rules that protect your data

Strongly disagree Disagree Neutral Agree Strongly agree Don't know
  • Minimum 5 points to avoid serious information loss.
  • Add a "Don't know" option to separate non-response from true neutrality.
  • Use parametric statistics only when points are roughly equally spaced.
Carry forward

What to take into the next section

  • The four cognitive steps, understanding, retrieval, judgement, and communication, are a checklist for evaluating any item you write.
  • Prefer continuous data capture over pre-set ranges; use ranges only for sensitive quantities.
  • Each closed-question type has its own design rules: checklists, multiple-choice, rating, and ranking all differ.
  • Rating scales need at least five points and a dedicated Don't know option.

Overview

Section 1 covered the up-front planning that shapes the entire instrument. This section zooms in on the individual question, and in particular on the choice between open-ended and closed-ended formats, arguably the most consequential design decision after objective-setting. It begins with what happens cognitively when a respondent answers any question at all, because that process explains the design rules that follow. It then covers open questions, the four main types of closed question and the trade-off between the two formats, and it ends with the rule that the categories of a multiple-choice question must be mutually exclusive and jointly exhaustive, the rule that draft items break most often.

Learning Objectives

  • 3.1 Describe the four cognitive steps involved in responding to a question, and identify the step at which a flawed item fails.
  • 4.1 Explain when and how to use open questions, including why exact numbers such as age are preferred to ranges and why income is the usual exception.
  • 4.2 Distinguish among the four main types of closed questions (checklist, multiple-choice, rating and ranking) and state the design rules for each.
  • 4.3 Identify the advantages and limitations of closed versus open question formats.
  • 5.1 Recognise multiple-choice categories that overlap or leave gaps, and repair them so that they are mutually exclusive and jointly exhaustive.

3.1 The Four Cognitive Steps of Answering a Question

When a respondent encounters a question, they go through four distinct cognitive steps. Understanding these steps, formalised in the cognitive model of the survey response process (Schwarz, 1999; Tourangeau & Yan, 2007), helps researchers design questions that produce accurate, reliable answers. Figure 3.1 draws the steps as a sequence in which a question can fail at any point.

The four-step cognitive model of survey response: understanding, retrieval, judgement, communication1Understandparse the question2Retrieverecall the facts3Judgeform an answer4Communicatemap to a responseerror can enter at any step
Figure 3.1. The cognitive model of the survey response process. A respondent moves through four steps, and a question can fail at any one of them: misreading the wording, failing to recall, judging poorly, or mismatching the available response options.

The first step is understanding: will the respondent understand the question? By this we mean that the respondent must take from the words the same meaning the researcher intended. If the wording is technical or ambiguous, different respondents will in effect answer different questions, so the question must be clearly worded in non-technical language. The second is retrieval: will the respondent know, or be able to recall, the answer? Even a question that is perfectly understood is useless if the information is not stored in the respondent's memory or is too hard to bring back. If the question requires information they do not have, they might skip it or fabricate a response, and a made-up answer is harder to detect than a blank one. The third is judgement: once the information is recalled, the respondent must turn it into an answer. If the question involves a subjective decision, such as an opinion or belief, each respondent may apply a different standard, so the designer should ask whether there is a way to make it less subjective. For example, asking how many days a person exercised in the past week requires less personal judgement than asking whether they are "active". Special care in question design is needed to capture such information reliably. The fourth is communication: the respondent must fit the answer they have formed into the options the questionnaire provides. This step asks whether the possible responses are clear, with an appropriate method of recording the answer, because a respondent who knows the true answer still cannot report it if no option matches.

Once you have drafted a question, these four steps become four questions to ask of it. Will the respondent understand it? Will they know the answer? If it requires judgement, can you make it more objective? Are the response options clear and appropriate? To diagnose a flawed item, read it as a respondent would and stop at the first step where an honest respondent would struggle, because that step tells you what to fix. Worked Example 3.1 applies this routine to one faulty item for each of the four steps; for each item, watch for the first point at which an honest respondent would get stuck and how the fix targets that point.

Worked Example 3.1: At Which Step Does the Item Fail?

  • Understanding. “Have you had an episode of acute gastroenteritis this month?” Many respondents will not know the medical term. Fix: use plain words, such as “Have you had diarrhea or vomiting since 1 March?”
  • Retrieval. “How many cups of coffee did you drink on each day of the past year?” Nobody can recall a daily count over a year, so the respondent skips the item or makes up a number. Fix: shorten the reference period, such as “On a typical day in the past 7 days, about how many cups of coffee did you drink?”
  • Judgement. “Is your diet healthy?” Each respondent must decide privately what “healthy” means, so answers are not comparable. Fix: ask about a concrete behaviour, such as the number of days in the past 7 on which the respondent ate vegetables.
  • Communication. “How often do you exercise? (Yes / No)” The respondent may know the answer, but the options cannot record a frequency. Fix: offer response options that match the question, such as a count of days in the past 7.

In general, a question that asks for too much detail over too long a period fails at retrieval, and the remedy is a shorter reference period or a “typical day” in a recent week.

The example shows that naming the failing step is what points to the fix: plain words for understanding, a shorter reference period for retrieval, a concrete behaviour for judgement and matching options for communication. These four steps apply to every question, whatever its format, and the rest of this section turns to the formats themselves, beginning with the open question.

4.1 Open Questions

Open questions (also called open-ended questions) place no restrictions on the type of response expected. They are more commonly used in qualitative research, because the responses may not be easily standardised for statistical analysis, but they can be valuable in quantitative research as well. Their most important use there is the fill-in-the-blank question for numerical data, and one rule governs it: capture an exact number wherever respondents will give one, because continuous data can always be grouped later but a range can never be turned back into an exact value. Income is the usual exception, because people are more willing to choose a range than to state an exact figure. Expand the four panels below for the details of each use of open questions in quantitative research, including how to word an income item and why the units must always be stated.

Fill-in-the-Blank Questions▼

One common type of open question in quantitative research is the fill-in-the-blank question, used especially for capturing numerical data. For example: “How many people live in this household? ___”

Where possible, capture numerical data as a continuous variable rather than using pre-defined ranges. Knowing someone is exactly 47 years old is more informative than knowing they fall in the 40–59 age bracket. Continuous data can always be categorised later during analysis. By this we mean that an exact age can be sorted into whatever groups the analysis needs, whether 40–59, 45–54 or simply "over 50", whereas once a respondent has ticked "40–59" there is no way to recover whether they were 41 or 58. Collecting the exact value therefore keeps every later option open.

Capturing Sensitive Data▼

When seeking sensitive information (such as total family income), respondents may be more willing to indicate a category or range rather than an exact value. In these cases, using ranges is an acceptable compromise to encourage honest responses.

Income is therefore the usual exception to the preference for exact numbers. Asking “What was your household’s income last year? $____” invites refusals and guesses. A better item asks which range best describes the income, offers ranges that do not overlap and together cover every possible income, and ends with “Don’t know” and “Prefer not to answer”. Worked Example 3.5 in Section 4 shows a complete example, and income items belong near the end of the questionnaire with the other sensitive questions.

Specifying Units and Scales▼

When capturing numerical data, always specify the units being used (kilograms, pounds, centimetres, etc.) and consider offering a choice of measurement scales to accommodate different respondents. The reason is that a bare number is uninterpretable without its unit. For example, a recorded weight of 70 could mean 70 kilograms or 70 pounds, which describe very different people, and the analyst cannot tell which was meant. Because some respondents know their weight in pounds and others in kilograms, letting them choose the unit, and recording which one they chose, avoids both guessing and conversion errors.

Comments Sections▼

Sometimes a “comments” section can be attached to a closed question to allow respondents to elaborate or express their opinion. This combines the advantages of both question types and can capture unexpected information.

Open questions therefore earn a place even in a structured questionnaire, both for exact quantities and, through a comments section, for answers the designer did not anticipate. Most items in a quantitative questionnaire, however, are closed questions.

4.2 The Four Types of Closed Question

In closed questions (also called closed-ended questions), the respondent selects from a pre-defined range of options. Closed questions are generally easier for respondents to answer and easier to code for data entry. They come in four main types, and each follows its own design rules: a checklist allows several answers, a multiple-choice question allows only one, a rating question records an opinion or judgement on a scale, and a ranking question records an order of priority. Open each tab below to see an example of each type and the rules for designing it.

Checklist Questions

In a checklist question, the respondent checks all options that apply. The options do not need to be mutually exclusive or jointly exhaustive. This is because each option is treated as a separate yes/no variable in the database. By this we mean that a list of five symptoms produces five columns in the dataset, one per symptom, and each respondent is coded "yes" or "no" in every column. A respondent who ticks both headache and fever simply has "yes" in two columns, so an overlap between options causes no problem.

Example: “Which of the following symptoms have you experienced? (Check all that apply)”, followed by a list of symptoms with checkboxes.

This is equivalent to having a series of individual yes/no questions for each category.

Two-Choice and Multiple-Choice Questions

Multiple-choice questions require categories that are mutually exclusive (no overlap) and jointly exhaustive (cover all possibilities). Adding an “Other (please specify)” category ensures exhaustiveness.

It is recommended to limit choices to 5 options for in-person/telephone interviews and 10 for mailed/internet questionnaires. The limit is lower when questions are spoken because a listener must hold every option in memory, whereas a reader can look back over a written list. In addition to the number of options, be aware of order effects. By this we mean that the position of an option in the list can change how often it is chosen, regardless of its content. In written, self-administered lists respondents tend to favour options near the top (a primacy effect), because they read from the top and may stop once they reach a plausible answer. In questions read aloud, by contrast, they tend to favour the last options they hear (a recency effect), because those options are freshest in memory when the interviewer stops speaking. Both, together with acquiescence (the habit of simply agreeing), are forms of satisficing, where a respondent settles for a good-enough answer rather than the most accurate one (Krosnick, 1991). Satisficing is not dishonesty; it reflects the fact that working out the most accurate answer takes effort, and a tired or uninterested respondent takes a shortcut. Varying the option order across different versions of the questionnaire can help mitigate these effects, because any advantage of being first or last is then spread across all the options rather than always favouring the same one.

Example: “How do you draw water from the cistern? (Select one)”, with options like “With a bucket,” “With a manual pump,” “With an electric pump,” “Other (please specify).”

Breaking this rule is the most common fault in multiple-choice items, and 5.1 shows how to test categories and repair them.

Rating Questions

Rating questions ask the respondent to assign a value on a pre-defined scale. The most common form is the Likert scale (Likert, 1932), where respondents indicate their level of agreement (e.g., strongly agree, agree, neutral, disagree, strongly disagree). Scale design choices, including number of points, midpoint, and label wording, affect both reliability and the likelihood that respondents will satisfice (settling for a satisfactory rather than an optimal answer; Krosnick, 1991).

Key considerations:

  • Use a minimum of 5 to 7 categories to avoid serious loss of information.
  • Decide whether to include a neutral midpoint or use a “forced-choice” scale (even number of categories). A midpoint such as “neutral” gives a place to respondents who genuinely have no view, but it also offers an easy resting place for those who are satisficing. A forced-choice scale removes that resting place by making every respondent lean one way or the other, at the cost of pushing truly neutral people into a side they do not hold.
  • Always include a “don’t know” or “not applicable” option to distinguish non-responses from true neutrality. By this we mean that a person who has thought about an issue and feels neither for nor against it is different from a person who has no knowledge of the issue at all. Without a “don’t know” option, both would tend to choose the midpoint, and the two very different answers would be recorded as the same. Krosnick (1991) noted that a respondent who is satisficing may also say “don’t know” to avoid the effort of forming an opinion, so a large share of “don’t know” answers is worth examining in its own right.
  • Parametric statistics (mean, standard deviation) should only be computed if there are at least 5 points on the scale and the points can be assumed to be equally spaced. A mean treats the step from “disagree” to “neutral” as the same size as the step from “neutral” to “agree”. If respondents do not experience the steps as equal, an average of the coded values can describe the group misleadingly.

A Visual Analogue Scale (VAS) is a special type of rating question where respondents mark a point on a continuous line between two extremes (e.g., “terrible” to “excellent”). VAS is widely used for pain measurement in clinical research.

Ranking Questions

Ranking questions ask respondents to order options by priority or importance. They can be difficult to complete, especially when the list is long, because all categories must be held in mind simultaneously.

Important limitations:

  • Rank intervals are unknown: the difference between ranks 1 and 2 may not equal the difference between ranks 2 and 3. For example, a respondent may care a great deal about their first and second priorities and hardly at all about their third, yet the ranks record only the order, not how far apart the items are in importance.
  • Respondents may assign “tied” ranks, and decisions about how to handle these must be made in advance.
  • If categories are omitted, it may influence how respondents rank the remaining options.
  • For in-person interviews, using physical cards that respondents can sort may simplify the ranking process.

Choosing among the types starts with two questions: should the respondent give one answer or several, and does the study need an amount, an opinion or an order of priority? Once the type is chosen, its rules follow, such as at least 5 points and a “don’t know” option for a rating scale. Whatever the type, a closed question trades some richness for efficiency, which raises the question of when an open format is the better choice.

4.3 Weighing Open Against Closed Formats

The choice between open and closed formats is a trade-off between richness and efficiency. Open questions capture richer, more detailed data in the respondent’s own words, including answers the designer did not anticipate, but they are harder to analyse because the answers must be coded afterwards. Closed questions are efficient to code and analyse, but they risk oversimplifying issues or forcing respondents into categories that do not fully represent their views (Krosnick, 1999). The best questionnaires often use a thoughtful combination of both: closed items for the variables being formally measured, and open elements, such as an “Other (please specify)” line or a comments section attached to a closed question, to catch what the closed options miss. When comparing the formats for a particular study, link the choice to its purpose, since exploration favours open questions while measuring and comparing favour closed ones.

When a closed format is chosen, it works only if its categories fit every respondent, and the rule for multiple-choice categories is where draft items most often fail.

5.1 Mutually Exclusive and Jointly Exhaustive Categories

For a multiple-choice question to work, every respondent must find exactly one category that fits. Two properties guarantee this, and they are worth committing to memory.

The Rule for Multiple-Choice Categories

The categories must be mutually exclusive (no overlap, so that no value fits two categories) and jointly exhaustive (covering all possibilities, so that every value fits one). Adding an “Other (please specify)” category helps to ensure exhaustiveness.

A checklist is exempt from the rule, because each of its options is a separate yes/no variable, but every multiple-choice item must pass it. The quickest check is to test the boundary values and the extremes of the range. Worked Example 3.2 applies that check to a common faulty age item; watch for the values that fit two categories, the values that fit none, and the final check that the repair has not created a new fault.

Worked Example 3.2: Repairing Overlapping and Incomplete Categories

A draft questionnaire asks: “What is your age? ☐ 18 to 30 ☐ 30 to 45 ☐ 45 to 65”.

Step 1, test the boundaries. A 30-year-old or a 45-year-old fits two categories, so the categories are not mutually exclusive.

Step 2, test the extremes. A 17-year-old or a 70-year-old fits none, so the categories are not jointly exhaustive.

Step 3, repair the item. The better item asks “How old are you, in years? ____”. An exact age can be grouped later in any way the analysis needs, which is the preference for exact numbers from 4.1. If categories must be used, make them non-overlapping and complete for the population surveyed, such as, for a survey of adults, 18 to 29, 30 to 44, 45 to 64, and 65 or older.

Step 4, check the repair. Test the new boundaries again: 29 and 30, 44 and 45, and 64 and 65 each fall in exactly one category, and the open top category leaves no gap above 64.

The example shows that one item can break both properties at once, so a repair that removes only the overlap, or only the gap, fixes half the problem. Testing boundaries and extremes, before and after the rewrite, is the routine to use on any multiple-choice item.

This section has described the four cognitive steps behind every answer and the design rules for open and closed questions. The key takeaways below summarise these points, and the next section turns to the wording, structure and pre-testing of questions.

Key Takeaways

  • 3.1 Every time someone answers a question, they move through four cognitive steps: understanding what is asked, retrieving the information needed, making a judgement when the question asks for an opinion or estimate, and communicating the answer through the response options you offer. A well-designed question supports all four steps, and a flawed one breaks down at one of them, most often retrieval or judgement. To diagnose a draft item, read it as a respondent would and stop at the first step where an honest person would struggle: a medical term such as “acute gastroenteritis” fails at understanding, a daily count of coffee over a whole year fails at retrieval, “Is your diet healthy?” fails at judgement, and Yes/No options for “How often do you exercise?” fail at communication. Naming the step tells you what to fix; for a retrieval failure, for example, the remedy is a shorter reference period or a “typical day” in a recent week. In an exam answer, name the step, explain briefly why the respondent would struggle there, and offer a rewrite that removes the problem.
  • 4.1 Open questions place no limits on how someone can answer, which makes them the natural choice for exploratory work and for capturing answers you did not anticipate, for example through a comments section attached to a closed question. In quantitative questionnaires their most useful form is the fill-in-the-blank, such as “How old are you, in years? ____”, which records an exact number rather than a pre-set range. Exact numbers are preferred because you can always group continuous data into categories later, but you can never recover the precision lost by asking for a range: knowing someone is 47 tells you more than knowing they are in the 40 to 59 bracket. Income is the usual exception, because people are more willing to choose a range than to state an exact figure, so a good income item offers non-overlapping ranges that cover every possible income, followed by “Don’t know” and “Prefer not to answer”. When you write a numerical item, also state the units (kilograms or pounds, for example), and when you see a sensitive quantity such as income asked as an exact amount, flag it as the exception and rewrite it in ranges.
  • 4.2 Closed questions give respondents a fixed set of options, and each of the four main types follows its own rules. In a checklist (“check all that apply”) the options may overlap, and each option becomes a separate yes/no variable in the dataset. A multiple-choice question allows only one answer, so its categories must be mutually exclusive and jointly exhaustive, usually with an “Other (please specify)” option, and it works best with no more than about 5 options in an interview and 10 on a mailed or online form. A rating question, most often a Likert scale, needs at least 5 points and a “don’t know” or “not applicable” option so that genuine uncertainty or non-response is not confused with a neutral answer, and means and standard deviations make sense only when the points can be assumed to be equally spaced. A ranking question asks people to put options in order, which is demanding for long lists and produces ranks whose intervals are unknown, so the gap between first and second may not equal the gap between second and third. When you choose a format, ask whether respondents should give one answer or several and whether you need an amount, an opinion or an order of priority, and then apply the rule for that type.
  • 4.3 Choosing between open and closed formats is a trade-off between richness and efficiency. Open questions capture detailed and unexpected answers in the respondent’s own words, but those answers have to be coded afterwards and are harder to analyse statistically. Closed questions are quicker to answer and easy to code and analyse, but they can oversimplify an issue or force people into categories that do not fit their situation. Well-designed questionnaires usually combine the two, using closed items for the variables being formally measured and open items, such as an “Other (please specify)” line or a comments box, to catch what the designer did not anticipate. When asked to compare the formats, give one advantage and one limitation of each and link your choice to the purpose of the study: exploration favours open questions, while measuring and comparing favour closed ones.
  • 5.1 For a multiple-choice question to work, every respondent must find exactly one category that fits. That means the categories must be mutually exclusive, so that no value fits two categories, and jointly exhaustive, so that every possible value fits one; an “Other (please specify)” option helps with the second rule. The quickest check is to test the boundary values and the extremes: in “18 to 30 / 30 to 45 / 45 to 65”, a 30-year-old fits two categories, while a 17-year-old or a 70-year-old fits none. To repair the item, ask for the exact number where you can (“How old are you, in years? ____”), or, if categories are needed, make them non-overlapping and complete for the population surveyed, such as 18 to 29, 30 to 44, 45 to 64 and 65 or older. In an exam answer, name both faults when both are present, give a value that shows each one, and check that your rewrite does not create a new gap or overlap.
Knowledge Check: this section

1. A draft item for a study of physical activity reads: “On each day of the past 6 months, how many minutes did you spend walking?” At which cognitive step is this item most likely to fail?

“Walking” is a plain, everyday word, so understanding is not where this item fails. The difficulty lies in what it asks respondents to remember, not in what the words mean.
A fill-in number of minutes can be recorded on a form, so communication is not the problem. The answer could be written down if anyone could recall it.
Judgement failures arise when an item asks for a subjective decision, such as an opinion or a vague word like “healthy”. Here the facts exist but are beyond memory, which is a retrieval problem even if respondents end up guessing.
Retrieval requires the respondent to recall the needed information, and nobody can recall a separate figure for each of about 180 days. A short, recallable reference period, such as a typical day in the past 7 days, repairs the item.
Correct answer: retrieval. The information needed is what the item asks the respondent to remember: a separate count of minutes for every day over six months, about 180 days. The words are plain, so understanding is not the problem, and a fill-in number can record the answer, so communication is not the problem either; the difficulty is that nobody can recall a daily figure that far back, so respondents skip the item or make up a number. The fix is a short, recallable reference period, such as “On a typical day in the past 7 days, about how many minutes did you spend walking?” Judgement is tempting because the respondent ends up guessing, but a judgement failure arises when the item asks for a subjective decision (an opinion, or a vague word such as “healthy”), not when the facts are beyond memory. (Section 2, 3.1 The Four Cognitive Steps of Answering a Question, Figure 3.1 and Worked Example 3.1; Objective 3.1)

2. A draft household questionnaire asks each adult’s age with the categories ☐ Under 20 ☐ 20 to 39 ☐ 40 to 59 ☐ 60 or older, and asks for household income as an exact dollar amount, $______. Which advice for these two items follows the lesson?

This keeps two faults. Age categories throw away detail that an exact age would keep, since exact values can be grouped later but ranges cannot be reversed, and an exact income amount invites refusals and guesses.
Exact age keeps full detail and can be grouped later into any categories, while income is the usual exception: because it is sensitive, non-overlapping ranges with “Don’t know” and “Prefer not to answer” reduce refusals.
Exact values usually do carry the most information, which is right for age, but “always” overlooks the exception for sensitive quantities. An exact income item leads many respondents to refuse or guess.
Ranges help with sensitive items such as income, but not with every question. Age is rarely sensitive, so banding it only loses detail that an exact number would keep and could later be grouped.
Correct answer: exact age, banded income. The information needed is the content of each item: age is rarely sensitive, whereas income is. Exact age is preferred because it can be grouped later into whatever categories the analysis needs, while a range can never be turned back into an exact value; income is the usual exception, because respondents who would refuse or guess an exact figure will often choose a range, so a banded item with “Don’t know” and “Prefer not to answer” reduces refusals. Asking both as exact numbers is tempting because it applies the general rule, but it forgets the exception and leaves the income item inviting refusals. (Section 2, slide “Capture continuous data wherever possible” and 4.1 Open Questions, Fill-in-the-Blank Questions and Capturing Sensitive Data panels; Objective 4.1)

3. A clinic questionnaire asks: “Which of the following symptoms have you had in the past 7 days? Tick all that apply: ☐ Cough ☐ Fever ☐ Sore throat ☐ Runny nose.” What type of closed question is this, and how should it be entered in the dataset?

A multiple-choice item allows one answer from mutually exclusive categories. “Tick all that apply” lets a patient choose several, and a single variable could not record someone with both a cough and a fever.
Nothing in the item asks patients to order the symptoms, and ranks are not equally spaced: the lesson notes that the gap between ranks is unknown, so they should not be stored or treated as equal-interval scores.
“Tick all that apply” makes this a checklist: the options need not be mutually exclusive, and each symptom is entered as its own yes/no variable, as if it were a separate yes/no question.
A rating item asks respondents to place an answer on a scale, such as satisfaction, and the five-point rule belongs to rating scales. This item asks only whether each symptom was present, which is a checklist.
Correct answer: a checklist, with each symptom as its own yes/no variable. The information needed is the instruction “Tick all that apply”, which lets a patient choose several options. That makes it a checklist question: its options need not be mutually exclusive, and because one patient can have both a cough and a fever, each option is stored as a separate yes/no variable, exactly as if it were a series of yes/no questions. Calling it multiple-choice is the tempting mistake: a multiple-choice item allows one answer from categories that are mutually exclusive and jointly exhaustive, so a single variable could not record a patient with two symptoms. (Section 2, slide “Four types, four sets of rules” and 4.2 The Four Types of Closed Question, Checklist tab; Objective 4.2)

4. A draft item reads: “How many times did you visit a doctor in the past 12 months? ☐ 1 to 3 ☐ 3 to 5 ☐ 6 to 10”. Which statement correctly describes the fault in the response categories and a repair?

Testing each possible answer shows both faults: 3 visits fits two boxes, so the categories are not mutually exclusive, and 0 or more than 10 visits fits none, so they are not jointly exhaustive. An exact count, or complete non-overlapping bands, repairs both.
The overlap is real and changing “3 to 5” to “4 to 5” removes it, but the categories still leave gaps: there is no box for 0 visits or for more than 10, so they are not jointly exhaustive.
Asking respondents with no visits to skip the item makes their answer indistinguishable from a missing one. Categories should be jointly exhaustive, so 0 needs its own box, and the overlap at 3 remains in any case.
Letting respondents choose either box means the same answer can be recorded in two ways, so the categories are not mutually exclusive and the counts cannot be interpreted. The item also leaves no box for 0 or more than 10 visits.
Correct answer: the categories both overlap and leave gaps. The information needed is the list of boxes: test each possible answer and ask whether it fits exactly one box. A respondent with 3 visits fits two boxes, so the categories are not mutually exclusive, and a respondent with 0 visits or with 12 visits fits none, so they are not jointly exhaustive; the repair is to ask for the exact number, or to use non-overlapping categories that cover every value, starting at 0 and ending with an open top category. Changing “3 to 5” to “4 to 5” is tempting because it removes the overlap, but it still leaves no box for 0 visits or for more than 10, so it fixes only one of the two faults. (Section 2, slide “Four types, four sets of rules” and 5.1 Mutually Exclusive and Jointly Exhaustive Categories, Worked Example 3.2; Objective 5.1)

5. A team evaluating a new walk-in clinic wants to ask patients, “What is the main reason you chose this clinic today?” The team needs to count how often each reason is given, but it is not sure it can foresee every reason. Which question format best balances the strengths and weaknesses of open and closed questions?

A closed list is quickest to code, but without an “Other” option it is not jointly exhaustive. Patients whose reason was not foreseen must skip the item or choose a category that is wrong.
An open question does capture every reason, but the claim that it needs no coding is wrong: open answers must be coded afterwards by hand before the team can count how often each reason is given.
Combining a closed list with “Other (please specify)” keeps most answers pre-coded and easy to count while still recording reasons the team did not anticipate, which also makes the list jointly exhaustive.
The team asked for the main reason, which is one answer, not an order of ten. Ranking is cognitively demanding with long lists and produces ranks with unknown intervals, and it still cannot capture a reason missing from the list.
Correct answer: a closed list plus “Other (please specify)”. The information needed is the team’s two goals: counting reasons favours closed answers, which are efficient to code and analyse, while not knowing every reason favours open answers, which capture what the designer did not anticipate. Combining the two keeps most answers in pre-coded categories, lets unexpected reasons be written in, and makes the categories jointly exhaustive. The closed list without “Other” is tempting because it is the easiest to code, but patients whose reason is missing must skip the item or pick a category that is wrong; an open question alone does capture every reason, but each answer must then be coded by hand. (Section 2, 4.3 Weighing Open Against Closed Formats, the Comments Sections panel in 4.1 Open Questions and the Multiple-Choice tab in 4.2; Objective 4.3)

✦ Pass the knowledge check with 100% to continue

Section 3

Wording, Structure & Pre-Testing

⏱ Estimated reading time: 14 minutes

Section 3 of 4

Wording, Structure & Pre-Testing

Clear, unbiased wording; questionnaire structure; validation before deployment.

Wording rules

Four rules that protect validity

Short and simple

Under 20 words; no jargon; match the respondent's vocabulary, not yours.

Specific

Name the time frame and clearly define the event. "In January" not "recently."

No double-barrelled phrasing

One construct per question. "Effective and trustworthy" is two separate questions collapsed into one.

No leading phrasing

Neutral framing only. The question should not imply which answer is expected.

Questionnaire structure

The funnel approach

Broad, general questions Topic-specific items Sensitive items

Start with the general; move toward the specific. Sensitive questions come last, after the respondent is already engaged.

Pre-testing

Four methods, each catching different problems

Expert review

Colleagues check coverage, clarity, and logic. Catches obvious problems early.

Field pre-test

Small sample from the target population. Reveals comprehension failures in practice.

Think-aloud (cognitive interview)

Respondents narrate their reasoning. Surfaces interpretation failures invisible to the designer.

Test-retest

Same respondents, two occasions. Assesses repeatability; only valid if the questionnaire is unchanged between administrations.

Carry forward

What to take into the next section

  • Four wording rules prevent the most common item-level biases: short, specific, no double-barrelled phrasing, no leading phrasing.
  • Structure questionnaires with a funnel within each section; sensitive items go last.
  • Pre-testing is mandatory. Expert review, field testing, think-aloud, and test-retest each catch different classes of problems.
  • Validation compares instrument responses against a gold standard or established method. Poor reliability attenuates associations in analysis.

Overview

Sections 1 and 2 settled the larger questions of planning and question format. This section is about the small choices that determine whether respondents understand a question the same way you do. Wording, structure, layout and pre-testing are where most of the residual measurement error in surveys is introduced, or eliminated. The section sets out four rules for wording a question and a routine for using them to repair a draft item, then the structure and layout of the questionnaire as a whole, and finally the pre-testing and validation that check whether the finished instrument works and measures what it should.

Learning Objectives

  • 5.2 Apply the four wording rules (short and simple, specific, not double-barrelled, not leading) to name the rule a draft item breaks and rewrite it.
  • 6.1 Describe how to structure and lay out a questionnaire: an introduction, easy opening items, the funnel approach, sensitive items near the end and pre-coded responses.
  • 6.2 Explain the importance of pre-testing and validation, and state what expert review, field pre-testing, think-aloud interviews, test-retest reliability and comparison with a gold standard can each reveal.

5.2 Four Rules for Wording Questions

The way a question is worded has a major impact on the validity of the results. Poorly worded questions can confuse respondents, introduce bias, or yield uninterpretable data (Schwarz, 1999). Four rules guard against these problems: keep each question short and simple, make it specific, avoid double-barrelled questions that ask about two things at once, and avoid leading questions that suggest a desired answer. Expand each rule below for its details and an example of an item that breaks it.

Rule 1: Keep It Short and Simple▼

Questions should rarely exceed 20 words. Avoid abbreviations, jargon, and complex technical terminology. Always consider the respondent’s level of technical knowledge.

Poor: “How many cases of acute gastrointestinal illness occurred during the time period?”

Better: “How many times did you have diarrhea during January?”

Rule 2: Be Specific▼

Make questions as specific as possible. If asking about a health event, specify the time frame (e.g., “over the past month”) and clearly define the event. For example, define what constitutes a “new case” of diarrhea: perhaps at least 2 days of normal bowel movements before onset counts as a new episode.

Choose a reference period that respondents can actually remember. A recent, short window (the past 7 days, or a typical day in the past week) gives more accurate answers than a daily count over a month or a year, which fails at the retrieval step described in Section 2.

Rule 3: Avoid Double-Barrelled Questions▼

A double-barrelled question asks about two things at once. For example: “Do you think traveller’s diarrhea is a serious problem that people should get vaccinated for?” is really asking two questions: one about the seriousness of the disease and one about vaccination. The problem is that a single answer cannot tell you which part the respondent was responding to. For example, a person who thinks the disease is serious but does not want vaccination has no honest way to answer, and if they answer “no”, the researcher cannot tell whether they rejected the seriousness, the vaccination, or both. These should therefore be separated into two distinct questions, each of which can be answered on its own.

Rule 4: Avoid Leading Questions▼

Leading questions suggest a desired answer and can bias responses. By this we mean that the wording signals which answer the researcher expects or approves of, and many respondents, wishing to be agreeable, move toward that answer. Because the push is always in the same direction, the error does not cancel out across respondents; it shifts the whole result. For example: “Should people have to suffer from regular bouts of diarrhea because of a bad water supply in this region?” implies that the answer should be “no.”

A more neutral alternative: “Do you think the water supply in this region needs to change?”

Two of the rules call for a short procedure rather than a single change. Being specific means naming the time frame and also defining what counts as one occurrence of the event, so that every respondent counts episodes the same way. Without that definition, one long illness can be counted as a single episode by one respondent and as several by another. The definition goes into the item itself as a counting instruction, for example “Count a new time only after at least 2 days of normal bowel movements.” Neutralising a leading item also takes a few steps. Remove the loaded premise or emotive wording (such as “suffer” or “Given the well-known dangers of ...”), and drop openings that presuppose the answer, such as “Do you agree that” or “Don’t you think”. Then either name both directions (“Do you support or oppose ...?”) or ask a plain open question such as “Do you think the water supply in this region needs to change?”, and offer balanced response options, for example from strongly support to strongly oppose. As with any repair, check that the rewrite does not add a new fault.

The rules are most useful as a routine for reviewing draft items, in three steps. First, read the item as a respondent and note the cognitive step at which you would struggle. Second, name the rule the item breaks, in a short phrase. Third, write an improved version and check that it removes the fault you named, without adding a new one. Worked Example 3.3 applies this routine to six draft items; watch for items that break more than one rule, for the episode definition written into the diarrhea item, and for the way the leading item is neutralised, because the rewrite must then remove every fault.

Worked Example 3.3: Diagnosing and Repairing a Draft Item

  • “Do you agree that the clinic’s staff are friendly and efficient?” Rules broken: double-barrelled (friendly and efficient are two judgements) and leading (“Do you agree that” invites agreement). Improved: two items, “How friendly were the clinic staff?” and “How efficient was the clinic’s service?”, each with a labelled five-point scale.
  • “Do you often feel tired?” Rule broken: not specific (“often” is vague and there is no time frame). Improved: “In the past 14 days, on how many days did you feel too tired to do your usual activities? ____ days”.
  • “How many episodes of nocturnal enuresis did your child experience?” Rules broken: not short and simple (medical jargon) and not specific (no time frame). Improved: “In the past 30 days, on how many nights did your child wet the bed? ____ nights”.
  • “How many cases of acute gastrointestinal illness occurred during the time period?” Rules broken: not short and simple (“acute gastrointestinal illness” and “cases” are jargon) and not specific (no time frame and no definition of an episode). Improved: “In the past 30 days, how many times did you have diarrhea? Count a new time only after at least 2 days of normal bowel movements. ____ times”.
  • “Should people have to suffer from regular bouts of diarrhea because of a bad water supply in this region?” Rule broken: leading (“have to suffer” and the premise about a bad water supply imply that the answer is “no”). Improved: “Do you think the water supply in this region needs to change?”, with the emotive premise removed and nothing in the wording that points to one answer.
  • “How many hours of television do you watch? ☐ 0 to 2 ☐ 2 to 4 ☐ 4 to 6” Rule broken: the categories are not mutually exclusive (2 and 4 hours fit two categories) or jointly exhaustive (more than 6 hours fits none), and there is no time frame. Improved: “On a typical day in the past 7 days, about how many hours of television did you watch? ____ hours”.

On an examination, an answer earns credit when it names the specific rule (not just “unclear”) and gives a rewrite that actually removes that fault.

The example shows that faults often come in pairs, and the last item also breaks the category rule from Section 2 (5.1), a reminder that wording and format problems occur together. Once each item is well worded, the items must be arranged in a logical order and presented on a usable form.

6.1 Structure and Layout of a Questionnaire

A well-structured questionnaire guides the respondent through a logical flow and creates a positive experience that encourages completion. Three elements frame that flow. The questionnaire should begin with an introduction that explains the rationale and importance of the questionnaire, how the data will be used and the assurance of confidentiality, and that estimates how long it will take to complete. The opening questions should build confidence: easy, non-threatening items that put the respondent at ease. The instructions should be clear and concise, with key instructions highlighted in bold or other formatting, remembering that people read instructions only when they think they need help.

Within the body of the questionnaire, questions should be grouped in sections, either by subject (illness, water supply, housing) or chronologically (current health, past week, past month, previous illness). Within each section, questions should follow the funnel approach, starting with broader, more general questions and gradually becoming more specific and focused, which helps respondents warm up to the topic before being asked for detailed information. For example, a section on water might first ask where the household gets its drinking water, then how the water is stored, and only then how often it is boiled before drinking. In addition, pairs of questions that capture essentially the same information can be included at different points to verify responses and check internal consistency. By this we mean that the same fact is asked twice in different ways, for instance age early in the form and year of birth later on. If the two answers agree, the researcher has more confidence in them; if they conflict, the response is flagged as possibly careless or misunderstood. Sensitive items, such as income, sexual behaviour or substance use, belong toward the end of the questionnaire. By then the respondent has built rapport and seen how the information is used, and a respondent who stops at a sensitive item has already answered everything before it. A questionnaire that opens with its most sensitive item reverses this logic and invites people to stop at the very start.

For mailed or online questionnaires, the layout of the form matters as well, because visual appeal and ease of completion are essential. Three considerations stand out: pre-coding the responses, a professional appearance and an optimal length. Click each card below to see what each involves and how it helps.

Pre-Code
Responses
Click to learn more
Professional
Appearance
Click to learn more
Optimal
Length
Click to learn more

Each of these makes the form easier to complete or to enter, and length returns in Section 4, because shorter questionnaires consistently achieve higher response rates. However carefully a questionnaire is worded, ordered and laid out, it can still contain problems that only testing reveals.

6.2 Pre-Testing and Validation

Every questionnaire must be checked before it is used, and the checking answers two different questions. Pre-testing asks whether the questionnaire works for real respondents, and validation asks whether it measures what it is intended to measure.

6.2.1 Pre-Testing Methods

All questionnaires must be pre-tested before deployment. Pre-testing identifies confusing, ambiguous, or misleading questions, layout problems, and issues with instructions, and it also serves to estimate completion time (Presser et al., 2004). Four methods are used, and each catches a different kind of problem. Open each tab to see what a method involves and what it reveals.

Expert Review

The first step is to have colleagues or subject-matter experts review the questionnaire. They can evaluate whether all important issues are covered and identify obvious problems with clarity or logic.

Field Pre-Test

Pre-testing on a small sample of the target population is essential. Have respondents complete the questionnaire as they would in the actual study. This reveals questions that are hard to understand, unanswerable, or that need additional response categories.

Think-Aloud Pre-Test

In a think-aloud pre-test, the respondent narrates their thought process as they work through each question, the core technique of cognitive interviewing (Beatty & Willis, 2007). This provides direct insight into how questions are being interpreted and where confusion or difficulty arises.

Test–Retest Reliability

If feasible, re-administer the questionnaire to the same group of respondents after an appropriate time interval. The interval should be long enough that they do not recall their original answers, but short enough that the underlying information has not changed. Both limits matter for the same reason: the comparison is meant to show whether the question itself produces the same answer twice. If the interval is too short, respondents may simply repeat what they remember writing, which makes the question look more repeatable than it is. If it is too long, real changes in the respondent's health or behaviour will make the answers differ even when the question is working well. This approach assesses the repeatability of the questions. Note: it is only valid if the questionnaire itself has not been modified between administrations.

Matching the method to the problem is the practical skill. Expert review checks coverage and logic from the designer’s side, a field pre-test shows which items real respondents cannot answer and how long the form takes, a think-aloud interview reveals how respondents actually interpret an item, and a test-retest comparison shows whether answers repeat. Repeatable answers are not necessarily correct ones, however, and that is the question validation addresses.

6.2.2 Validation and the Cost of Unreliable Measurement

Validation assesses whether the questionnaire actually measures what it is intended to measure, and four approaches can be used. The most direct is comparison with a gold standard, in which questionnaire responses are compared with directly measured quantities. A gold standard is the best available measure of the true value, one we are prepared to treat as correct. For example, dietary intake reported on a food frequency questionnaire can be compared against actual measured intake, such as food weighed and recorded over several days. If the two agree closely, the questionnaire can be trusted to stand in for the more costly direct measurement. Because a gold standard is often expensive or impossible to obtain, responses can also be compared with an established method, the results of a well-validated existing instrument. When no external standard exists, repeated administration of the questionnaire to the same respondents assesses its repeatability. Finally, method comparison evaluates whether different administration methods (e.g., mail vs. telephone) produce comparable results.

Repeatable Is Not the Same as Valid

Close agreement between two administrations shows that answers are repeatable, not that they are correct: respondents who always under-report a behaviour will under-report it both times, giving close agreement around the wrong value. Showing validity needs a comparison with a gold standard or an established instrument, and a test-retest comparison is valid only if the questionnaire is unchanged between administrations.

Reliability matters well beyond the questionnaire itself, because unreliable measurement distorts every analysis that uses it. Interactive 3.1 shows why. Picture any measurement as a true signal plus random noise. By this we mean that each recorded answer is made up of the respondent's real value plus a random error that pushes it a little up or a little down, for example because of a misread word or a hurried guess. Reliability is the share of the measured variation that comes from the true signal rather than the noise, so it runs from 0 (all noise) to 1 (no noise). As measurement error grows, reliability falls, and the associations you later estimate with that noisy variable are dragged toward zero. This happens because the random errors are unrelated to the outcome. They spread the measured values out without carrying any of the true relationship, so the real pattern linking exposure and outcome is diluted and looks weaker than it is. For example, if people who truly eat more salt have higher blood pressure, but reported salt intake is full of random error, some heavy salt users will be recorded as light users and the reverse, and the estimated link between salt and blood pressure will shrink. Epidemiologists call that systematic under-statement attenuation bias. One remedy is to add more items to a scale. Because the random errors on separate items tend to point in different directions, averaging several items cancels out some of the noise and pushes reliability back up. Move the sliders below and watch these effects happen together.

📊 Interactive 3.1: Reliability & Measurement Error

Generate 100 respondents with a "true" attitude score; measurements are corrupted by random noise. Watch test-retest correlation, Cronbach’s α (when 5 items are simulated), Cohen’s κ (when scores are dichotomized), and the attenuation of any downstream regression coefficient.

Measurement-1 vs Measurement-2
Effect of error on a regression slope

Outcome Y depends on the true X. The regression on the measured X is attenuated by error.

Reliability (ρ)
–
Cronbach α
–
Cohen κ
–
Attenuated β
–
Try this: hold true variance fixed, increase error. Reliability falls; the regression slope on the measured X visibly shrinks toward zero. That is attenuation bias. Adding more items (α mode) recovers reliability via the Spearman–Brown formula.

Interactive 3.1 makes the practical case for pre-testing and validation: a questionnaire with poor reliability does not merely produce noisier answers but also biases the associations estimated from them toward zero.

This section has set out the rules for wording questions, the structure and layout of a questionnaire, and the pre-testing and validation that precede its use. The key takeaways below summarise these points, and the next section turns to response rates and data coding.

Key Takeaways

  • 5.2 Small changes in wording can change the answers you get, so four rules guide every item you write. Keep it short and simple: usually under 20 words, in plain language, with no jargon or abbreviations. Be specific: name the time frame, define the event (for example, what counts as a new episode of diarrhea) and choose a reference period that people can actually remember. Ask about only one thing at a time, because a double-barrelled item such as “Do you agree that the clinic’s staff are friendly and efficient?” cannot be answered by someone who feels differently about each part, and avoid leading wording that hints at the expected answer, such as “Should people have to suffer from regular bouts of diarrhea because of a bad water supply in this region?” When you repair a draft item, work in three steps: name the specific rule it breaks (not just “unclear”), write an improved version, and check that the rewrite removes that fault without creating a new one, as the neutral alternative “Do you think the water supply in this region needs to change?” does for the leading example.
  • 6.1 The order and look of a questionnaire affect whether people finish it and how accurately they answer. Open with an introduction that explains the purpose, how the data will be used and how confidentiality is protected, and that estimates how long the questionnaire will take; then begin with easy, non-threatening questions that build the respondent’s confidence. Group related questions by subject or chronologically, and within each section use the funnel approach, moving from broad, general questions to more specific ones. Place sensitive items, such as income, sexual behaviour or substance use, near the end, when rapport has been built and a respondent who stops has already answered everything before it. For mailed and online forms, pre-code the responses by printing numerical codes beside the options, keep the layout professional, the instructions clear and the questionnaire as short as possible (ideally under about 1,000 words). When you critique a questionnaire’s structure, check whether it has an introduction, whether it opens with easy items, whether each section moves from general to specific, and whether sensitive questions come too early.
  • 6.2 Every questionnaire must be pre-tested before it is used, because problems that are invisible to the designer often become obvious once real people try to answer. Each pre-testing method catches something different: expert review by colleagues checks coverage, clarity and logic; a field pre-test on a small sample of the target population shows which items are hard to understand, go unanswered or need more response categories, and how long the form takes; a think-aloud interview, in which respondents narrate their reasoning as they answer, reveals how each item is actually interpreted; and test-retest administration to the same people on two occasions assesses repeatability, provided the questionnaire is not changed in between. Validation asks a different question: does the questionnaire measure what it claims to measure? The most direct approach compares answers with a gold standard, such as dietary intake from a food frequency questionnaire against directly measured intake, and where no external standard exists you can compare with an established instrument, repeat the administration or compare administration methods. On an exam, match the method to the problem described (for example, a think-aloud interview for an item that respondents seem to read in different ways), and keep the distinction clear: pre-testing checks that the questionnaire works, validation checks that it measures the right thing, and poor reliability drags the associations you later estimate toward zero.
Knowledge Check: this section

1. A draft item reads: “In the past 7 days, did you wash your hands and use hand sanitiser after riding the bus?” (Yes / No). Which rule of question design does it break?

A missing time frame is a common fault, but this item names one, the past 7 days, so it is specific about the period. Its fault lies in how many behaviours it asks about.
Mentioning hand sanitiser is not a persuasive premise or an expected answer; it is a second behaviour. A leading item pushes respondents toward an answer, as “Don’t you agree” would, which this item does not do.
The word “and” joins two behaviours, washing hands and using sanitiser, with room for one answer, so a respondent who did only one cannot answer accurately. Splitting it into two items repairs the double barrel.
The lesson’s guide is about 20 words, and this item has 18, so length is not its problem. It is short enough; the fault is that it combines two behaviours in one yes/no answer.
Correct answer: double-barrelled. The information needed is the word “and”, which joins two behaviours, washing hands and using sanitiser, in a single yes/no item. A respondent who washed but did not use sanitiser cannot answer accurately, and a “no” cannot be interpreted; the repair is two items, “In the past 7 days, did you wash your hands after riding the bus?” and “In the past 7 days, did you use hand sanitiser after riding the bus?” “Not specific” is tempting because a missing time frame is a common fault, but this item names one (the past 7 days), and at 18 words it is also within the 20-word guide. (Section 3, 5.2 Four Rules for Wording Questions, Rule 3 panel, and Worked Example 3.3; Objective 5.2)

2. A draft item reads: “Don’t you agree that the city should build more bike lanes to protect cyclists from injury?” Which rewrite removes the fault without adding a new one?

The question itself sounds neutral, but the statement before it about what most residents want pushes respondents toward agreeing. A leading premise placed before the question still leads.
This rewrite names both directions, drops the persuasive reason about protecting cyclists and offers a balanced five-point scale, so it removes the leading wording without adding jargon or a second topic.
Adding lower speed limits makes the item double-barrelled, since a respondent may support one measure and oppose the other, and “Do you agree that” still frames the question in one direction.
This rewrite swaps plain words for jargon such as “augment” and “mitigate”, breaking the rule on clear language, and it keeps the persuasive reason about injury that made the original leading.
Correct answer: “Do you support or oppose building more bike lanes in the city?” with a balanced five-point scale. The information needed is the fault in the draft: it is leading, because “Don’t you agree that” and the persuasive reason (“to protect cyclists from injury”) both push respondents toward “yes”. The repair names both directions, drops the persuasive reason and offers balanced response options, so it removes the fault without adding another. The first rewrite is tempting because its question sounds neutral, but the statement before it about what most residents want still pushes respondents to agree; the third adds a second topic (double-barrelled), and the fourth adds jargon while keeping the persuasive reason. (Section 3, 5.2 Four Rules for Wording Questions, Rules 1, 3 and 4 panels, and Worked Example 3.3; Objective 5.2)

3. A draft questionnaire for a sexual-health study has no introduction. It opens with “How many sexual partners have you had in the past 12 months?”, then asks about the use of sexual-health services, and ends with general questions about health. Which revised order follows the lesson?

This follows subsection 6.1: an introduction covering purpose, use of the data, confidentiality and time, easy non-threatening items first, a funnel from general to specific, and sensitive items near the end once rapport has built.
Opening with the most sensitive item raises the chance that people stop at the start, and those who stop lose all their other answers too. Sensitive items belong near the end, so a respondent who stops has answered everything before.
Marking the item optional does not change its position at the very start, before any introduction or rapport, and keeping general questions at the end reverses the funnel, which moves from general to specific.
The introduction is right, but placing the sensitive item straight after it still ignores the advice to begin with easy, non-threatening items and to place sensitive items near the end, after rapport has built.
Correct answer: introduction, easy opening items, general to specific, sensitive item near the end. The information needed is the draft’s order: it has no introduction and opens with its most sensitive item, the reverse of the recommended structure. Subsection 6.1 says to begin with an introduction that explains the purpose, how the data will be used, confidentiality and the time required, then easy, non-threatening questions; within each section the funnel moves from general to specific; and sensitive items go toward the end, after rapport has built, so that a respondent who stops there has already answered everything before it. Asking the sensitive item while respondents are “fresh” is tempting, but it raises the chance that people stop or refuse at the very start, which loses all of their other answers as well. (Section 3, slide “The funnel approach” and 6.1 Structure and Layout of a Questionnaire; Objective 6.1)

4. In a pre-test of a diet questionnaire, the team suspects that respondents read “servings of vegetables” differently: some may count a large salad as one serving and others as three. Which pre-testing method is designed to show how respondents interpret the phrase?

Test-retest shows only whether answers repeat. A respondent who consistently reads “serving” in their own way would give the same answer both times, so the method cannot reveal how the phrase is interpreted.
A gold standard such as weighed food records can show whether answers are accurate, but not why they are wrong, so it cannot reveal how respondents understand “servings of vegetables”.
Expert review checks coverage and logic from the designer’s side. Two nutrition researchers would bring their own definition of a serving rather than showing how ordinary respondents read the phrase.
In a think-aloud (cognitive) interview respondents narrate their reasoning as they answer, so the team hears directly how each person counts a serving, which is exactly the interpretation problem in question.
Correct answer: a think-aloud interview. The information needed is the kind of problem: respondents read the same words but may mean different things by them, which is a problem of interpretation. In a think-aloud pre-test (cognitive interview) respondents narrate their reasoning as they answer, so the team hears directly how each person counts a serving and can then define a serving on the form. Test-retest is tempting, but it shows only whether answers repeat, and a respondent who consistently reads “serving” in their own way will give the same answer both times; a gold standard shows whether answers are accurate without showing why they are not, and expert review checks coverage and logic from the designer’s side rather than the respondent’s. (Section 3, slide “Four methods, each catching different problems”, 6.2.1 Pre-Testing Methods (tabs) and 6.2.2 Validation and the Cost of Unreliable Measurement; Objective 6.2)

5. A draft item in a web questionnaire for undergraduates reads: “How often do you drink energy drinks and coffee?” Which rewrite removes every fault in the item?

Two separate items remove the double barrel, and each names a 7-day reference period with a count of days, so the vague “how often” is replaced by a specific time frame. Both faults are gone.
Splitting the item fixes the double barrel, but “How often” still has no time frame, so each new item remains not specific and respondents will each choose their own period.
Adding “in the past 7 days” fixes the time frame, but energy drinks and coffee still share one answer, so the item remains double-barrelled; a respondent who drank only coffee cannot answer accurately.
This version keeps the double barrel, still has no time frame, and adds a judgemental, leading phrase, “too often”, which asks for a subjective verdict instead of a count.
Correct answer: two items, each with a reference period and a count. The information needed is that the item breaks two rules: it is double-barrelled, because energy drinks and coffee are two behaviours with room for only one answer, and it is not specific, because “how often” has no time frame. A rewrite earns credit only if it removes both faults. Splitting the item into two is tempting because it fixes the double barrel, but without a time frame each new item is still vague; adding “in the past 7 days” to the single item fixes the time frame but leaves the double barrel, and the yes/no version adds a leading, judgemental word (“too often”). (Section 3, 5.2 Four Rules for Wording Questions, Rules 2 and 3 panels, and Worked Example 3.3; Objective 5.2)

6. To pre-test a questionnaire on sleep, a team gives it to 20 students twice, two weeks apart, and rewords one item between the two rounds. A colleague says: “If the answers agree closely, we will have validated the questionnaire.” Which response is correct?

The two-week interval is sensible, but the conclusion is wrong. Close agreement shows only that answers repeat; answers can be wrong in the same way both times, so validity needs a gold standard or an established instrument.
Close agreement validates no item, because test-retest assesses repeatability, not validity. The reworded item cannot even be assessed for repeatability, since the questionnaire must be unchanged between administrations.
Test-retest measures repeatability, while validity needs comparison with a gold standard such as directly measured sleep, and changing an item between rounds means any disagreement on it may reflect the new wording.
The lesson treats two weeks as sensible: long enough for students to forget their answers, short enough that sleep has probably not changed. A one-day gap would let them remember, and validity would still not be shown.
Correct answer: disagree, for two reasons. The information needed is what was done (the same questionnaire given twice, with one item changed) and what was claimed (validity). Giving a questionnaire twice measures repeatability: answers can agree closely and still be wrong in the same way both times, so validity needs a comparison with a gold standard or an established, validated instrument. Test-retest is also valid only if the questionnaire is unchanged between administrations, so any disagreement on the reworded item may reflect the new wording rather than unreliable answers. The two-week interval is tempting to criticise, but it is sensible: long enough for students to forget their answers, short enough that their sleep has probably not changed. (Section 3, 6.2.1 Pre-Testing Methods, Test–Retest tab, and 6.2.2 Validation and the Cost of Unreliable Measurement; Objective 6.2)

✦ Pass the knowledge check with 100% to continue

Section 4

Response Rates & Data Coding

⏱ Estimated reading time: 12 minutes

Section 4 of 4

Response Rates & Data Coding

Strategies for maximising participation, and principles for turning responses into clean data.

Why response rates matter

Non-response is a threat to validity

A low response rate is not merely inconvenient. If non-responders differ systematically from responders, your estimates are biased.

Design strategies

Clear purpose, professional layout, thorough pre-testing, minimal length, avoid unnecessary sensitive items.

Administrative strategies

Follow-up contact, incentives, stamped return envelopes, advance notice, personalised addressing, hand delivery.

One of the most effective strategies

Follow up with non-responders

Repeat contact with non-responders is one of the most effective ways to raise response rates.

  • Send reminders and re-deliver to non-responders.
  • Offer a small financial incentive.
  • Provide a stamped, first-class return envelope.
  • Notify participants in advance that a questionnaire is coming.
  • Personalise the cover letter; hand delivery helps where feasible.
Data coding

Three principles that protect data integrity

Explicit missing-value code

A unique value such as −999 for every missing field. Blank cells are indistinguishable from coder errors.

Consistent conventions

0 = No, 1 = Yes everywhere. Inconsistency produces silent errors in analysis.

Code on the form, then enter

Separate the coding and entry steps. Use a distinctive ink colour for codes to distinguish them from respondent answers.

Data entry software

Spreadsheets carry a specific risk

Spreadsheets

Easy to set up. But sorting a single column separates data from the same respondent across rows, silently destroying the dataset.

Specialised software

Enforces data integrity and validation rules. Safer for all but the smallest studies.

Storing Likert items as ordered categorical variables in statistical software protects every downstream analysis from type errors.

Carry forward

The full instrument lifecycle

  • Low response rates increase the risk of selection bias; follow-up contact is one of the most effective remedies.
  • Code missing values explicitly; never leave fields blank.
  • Maintain consistent coding conventions throughout the dataset.
  • Use specialised software for data entry; spreadsheets carry a structural risk that can destroy records silently.

Overview

Earlier sections covered the design of the questionnaire itself. This section turns to two operational concerns that can make or break a survey: getting people to respond at all, and turning their responses into a clean, analysable dataset. Both deserve as much attention as any decision earlier in the design process. The section explains why low response threatens validity and how to raise it, sets out the principles for coding responses and for deriving analysis variables from them, and ends with the practicalities of data entry and management.

Learning Objectives

  • 7.1 Explain why low response threatens validity, and identify design and administrative strategies for maximising response, including follow-up of non-responders.
  • 7.2 Explain best practices for coding questionnaire data, including a distinct code for missing answers, separate codes for “don’t know” and refusals, and consistent conventions.
  • 7.3 Describe considerations for computer data entry and management, including why coding and data entry are kept separate and why sorting one spreadsheet column can destroy a dataset.

7.1 Maximising Response

7.1.1 Why Low Response Threatens Validity

Regardless of the type of questionnaire used, researchers must make efforts to maximise the response rate. Low response rates increase the risk of selection bias: if non-responders differ systematically from responders, the study results may not accurately represent the target population (Edwards et al., 2009). For example, if people who smoke are less likely to return a health questionnaire than people who do not, the responders will contain proportionally fewer smokers than the population, and the survey will understate smoking prevalence. It is tempting to think that this can be fixed by simply collecting more questionnaires, but that is not the case. Collecting more replies from the same unrepresentative group would make an estimate more precise but not less biased. By this we mean that a larger number of replies narrows the confidence interval around the estimate, but the estimate stays centred on the wrong value, because every extra reply comes from the same kind of person who was already over-represented. This is why the share of the sample that responds matters, not only the number of completed questionnaires.

The term “response rate” refers to the proportion of study subjects who complete the questionnaire. Despite the name, it is a proportion rather than a true rate: a rate carries person-time in its denominator, whereas a response rate is one count divided by another. The label “response rate” is nonetheless universal and well understood in the research community. Read in reverse, the same proportion helps with planning: the number of people to invite is the number of completed questionnaires needed divided by the expected response rate, so 600 completed questionnaires at an expected response of 30% need 600 ÷ 0.30 = 2,000 invitations.

7.1.2 Strategies That Raise Response

Many strategies have been shown to improve response rates, many of them tested across hundreds of randomised trials in the Cochrane methodology review by Edwards et al. (2009). They fall into two groups: design strategies, which concern the questionnaire itself, and administrative strategies, which concern contact with respondents, incentives, and the delivery and return of the questionnaire. Of all of them, follow-up of non-responders with reminders and repeat deliveries is one of the most effective. Open each tab to see the strategies in each group.

Questionnaire Design Strategies

The first group of strategies concerns the questionnaire itself: its stated purpose, layout, pre-testing, length and content.

  • Make objectives clear: Explain the purpose of the study clearly to participants so they understand why their participation matters.
  • Professional layout: Ensure the questionnaire has a clear structure and professional appearance that signals credibility.
  • Pre-test thoroughly: A well-tested questionnaire is easier and less frustrating to complete. Provide an estimate of the time required.
  • Minimise length: Keep the questionnaire as short as possible; aim for under 1,000 words. Shorter questionnaires consistently achieve higher response rates.
  • Avoid sensitive questions: Unless absolutely necessary, do not ask for sensitive information that might deter participation or trigger social-desirability bias in those who do answer (Tourangeau & Yan, 2007).

Administrative Strategies

The second group concerns contact with respondents, incentives, and the delivery and return of the questionnaire.

  • Follow-up contact: Send reminders and repeat deliveries to non-responders. This is one of the most effective strategies.
  • Incentives: Offer small incentives for completion. Research shows that financial incentives help, though larger payments do not necessarily produce proportionally higher response rates.
  • Include a pen: Including a pen with a mailed questionnaire has been shown to significantly increase response rates (Sharp et al., 2006).
  • Return envelope: Provide a stamped, first-class return envelope (not business-reply).
  • Advance notice: Notify participants in advance that a questionnaire is coming.
  • University sponsorship: Mentioning university or institutional affiliation can increase perceived legitimacy.
  • Delivery method: Hand delivery or courier can improve response rates over standard mail.
  • Personalise: Address the questionnaire and cover letter personally to the respondent.
  • Ink colour: Coloured ink may have a small positive effect on response rates compared with standard black ink.

Several of the design strategies are the design rules of Section 3 seen from the respondent’s side. Worked Example 3.4 applies both groups of strategies to a mailed survey with a low response rate; watch which changes belong to design and which to administration, and which single step matters most.

Worked Example 3.4: The Low-Response Study

A research team mails a 15-page questionnaire to 2,000 households. The questionnaire uses small print, complex medical terminology, and includes a business-reply envelope. After 6 weeks, only 22% of questionnaires have been returned. What design and administrative changes would improve the response rate?

Step 1, design changes. Shorten the questionnaire well below 15 pages, use larger print and a professional layout, replace medical terms with plain language, and pre-test the revised form.

Step 2, administrative changes. One of the most effective steps is to follow up the non-responders with a reminder and a fresh copy of the questionnaire. A stamped, first-class return envelope, advance notice, a personalised cover letter and a small incentive also help.

Step 3, why it matters. With only 22% returned, the responders may differ systematically from the non-responders, so the estimates are at risk of selection bias.

The example shows that a strong answer to a low-response scenario combines both groups of strategies, includes follow-up of non-responders, and links the low response back to selection bias. A good response rate yields completed questionnaires, but they become usable data only once every answer has been turned into a number.

7.2 Coding Questionnaire Data

Before administering any questionnaire, researchers must plan how responses will be coded and entered into a database. Good coding practices ensure data integrity and make analysis more efficient. Some principles apply to every item, and some decisions concern how a single item becomes one or more analysis variables.

7.2.1 Coding Principles

Two principles govern the coding of every item: a unique code for missing values, with separate codes for “don’t know” and refusals, and consistent conventions throughout the questionnaire. Expand each panel below for the details and the mistakes each principle prevents.

Missing Value Codes▼

Designate a single, unique value to represent missing data (e.g., -999). This value should not be a legitimate answer to any question. Never leave missing values blank, because doing so makes it impossible to distinguish between items the respondent skipped and items the coder missed.

Give “don’t know” and “prefer not to answer” (a refusal) codes of their own as well, distinct from the missing code and from every real answer. A refusal, a genuine “don’t know” and a skipped item mean different things, and the panel “Non-Substantive Answers Need Codes of Their Own” in 7.2.2 explains why. Never code a missing answer as 0, because 0 can be a real answer and would be averaged in with the real values.

Consistency in Coding▼

Be consistent throughout the questionnaire. For dichotomous (yes/no) variables, always use the same coding convention (e.g., 0 = no, 1 = yes). This avoids confusion during data entry and analysis.

Codes for Answers That Are Not Answers

A skipped item, a “don’t know” and a refusal each get their own code, such as -999, 7777 and 8888, chosen outside the range of real answers. Never use a blank and never use 0, and declare the reserved codes missing in the statistical software before any analysis is run.

A third principle, coding on the form as a step separate from data entry, belongs with data entry in 7.3. Before that, the coding of a single item deserves a closer look, because one answer can become several different variables.

7.2.2 From Response Options to Analysis Variables

The form a question takes on the page and the form it takes in an analysis are two separate decisions. A single item will usually support several different variables, and the choice among them belongs to the analysis plan rather than to the questionnaire. Working that choice out in advance is part of designing the study, because it determines what has to be asked in the first place. An analyst can always collapse detail that was collected. No analyst can recover detail that was never collected.

Income illustrates the point. Section 2 (4.1) established that exact quantities are preferable to pre-set ranges wherever respondents will supply them, and that income is the standard exception, because refusal rates fall when income is asked in bands. Worked Example 3.5 follows one banded income item from the questionnaire into the dataset; watch how the same answer can become five different variables, each with its own use and its own cost.

Worked Example 3.5: One Income Item, Several Analysis Variables

The question as asked. “What was your total personal income, before taxes, in the past 12 months?” (Select one)

  • Less than $10,000
  • $10,000 to $19,999
  • $20,000 to $29,999
  • $30,000 to $39,999
  • $40,000 to $49,999
  • $50,000 to $59,999
  • $60,000 to $69,999
  • $70,000 to $79,999
  • $80,000 to $89,999
  • $90,000 to $99,999
  • $100,000 or more
  • Don’t know
  • Prefer not to answer

The variables it can produce. The respondent sees one question with eleven substantive options and two non-substantive ones. The dataset can hold that answer in any of several forms, and the study’s hypotheses determine which form is used. Table 3.2 sets out the most common possibilities for this single item.

Table 3.2. Five ways of coding the banded income item, with the variable each produces, its analytic use and its cost.

Coding decisionVariable producedAnalytic useWhat it costs
Retain the bands as askedOrdinal variable with 11 substantive levelsDescribing the distribution; testing for a trend across bandsSparse cells at the extremes, and ten coefficients in a regression model, each read against a single reference band
Collapse into lower and higher incomeBinary variable (0 = higher, 1 = lower)A single contrast that is easy to communicate and has adequate numbers in each groupEveryone within a group is treated as equivalent, and the cut point carries the entire result
Classify against a poverty thresholdBinary variable defined by an external standardQuestions about material deprivation, and comparison with studies using the same thresholdThe threshold varies with household size and region, so those items must also appear on the questionnaire
Assign the midpoint of each bandContinuous variable measured in dollarsRegression with one coefficient per $10,000 of income, and other parametric methodsAssumes respondents are evenly distributed within each band, and the open top band has no midpoint
Reserve values for non-substantive answers8888 = prefer not to answer; 7777 = don’t knowKeeps a refusal separable from a genuinely low income and from a coding omissionMust be declared missing in the statistical software before any analysis is run

None of these codings is correct in the abstract. Each answers a different question, and a well-planned study will often carry more than one of them in the same dataset, with the primary analysis pre-specified and the remainder reported as sensitivity analyses.

The example shows that the coding of an item is an analytic decision with consequences, not a clerical step. Six considerations should inform the choice among the codings in Table 3.2, from the one-way nature of information loss to the companion items that some analyses require. Expand each panel below to read one consideration.

Information Loss Runs in One Direction▼

Eleven bands can be collapsed into two categories at any point in the analysis, and the collapse can be revisited as often as the analyst wishes. Two categories can never be expanded back into eleven. The questionnaire should therefore record the finest distinctions respondents will tolerate, and the coarsening should be left to the coding stage, where it remains reversible. This is the principle behind the earlier recommendation to capture exact ages rather than age brackets, applied one step further along the pipeline.

Interpretability Falls as Categories Multiply▼

An eleven-level categorical predictor contributes ten coefficients to a regression model, each estimated against a single reference band and each carrying its own confidence interval. By this we mean that one band, often the lowest, is chosen as the baseline, and each of the other ten bands receives a coefficient describing how its outcome compares with that baseline. Few readers can hold ten contrasts in mind at once, and the cells at the extremes are often too small to produce stable estimates. Collapsing to three or four categories usually improves both precision and communication. The gain comes at a price, because everyone inside a category is treated as equivalent, so a respondent earning $31,000 and a respondent earning $89,000 become the same observation.

Cut Points Require a Justification That Precedes the Data▼

A threshold drawn from a poverty measure, a clinical guideline, or a published national category is defensible, because it refers to something outside the dataset and because other studies use it, which makes the estimate comparable. A threshold chosen because it produced the largest association is indefensible: the procedure guarantees an apparent effect, inflates the Type I error rate, and rarely replicates. The reason is that an analyst who tries many cut points and keeps the one with the strongest result is, in effect, running many tests and reporting only the most striking. Some cut point will show a large association by chance alone, even when income has no real effect, so the chance of a false-positive finding (a Type I error) rises well above the stated level. Because that chance pattern is specific to the sample, a new study using the same cut point usually fails to find it again. A median split sits between the two. It is convenient, but the threshold is a property of the sample rather than of the construct. By this we mean that the median income of one study's respondents may be $35,000 while the median in another study is $60,000, so “lower income” refers to quite different people in each, and two studies using median splits are not describing the same groups. The cut point and its rationale belong in the analysis plan, written before the data are examined.

Treating Bands as Numbers Imposes Assumptions▼

Midpoint coding assigns $15,000 to everyone in the $10,000 to $19,999 band, which assumes respondents are distributed evenly across it. Income distributions are right-skewed, so the assumption holds imperfectly, and it fails entirely for the open-ended top band, which has no midpoint. A common convention assigns 1.5 times the lower bound, although any value chosen there is an assumption that should be stated in the methods. Coding the bands as consecutive integers carries a stronger assumption still, that the distance between adjacent bands is constant, which fails whenever the band widths differ.

Non-Substantive Answers Need Codes of Their Own▼

A refusal is not a low income, and it is not a missed keystroke. Reserving distinct values, such as 8888 for a refusal and 7777 for “don’t know”, preserves information that matters, because refusal on an income item is patterned rather than random, and a concentration of refusals within one subgroup is a finding worth reporting.

Two cautions apply. Reserved values must be declared missing in the statistical software before analysis, because otherwise 8888 enters a model as a real observation and the error produces no warning. Reserved values must also lie outside the plausible range of the variable they occupy. The code 8888 is safely out of range for a banded variable numbered 1 to 11, but it is an entirely plausible income once the bands have been replaced by midpoints in dollars, so the same study may need different reserved values for the banded and the continuous versions of the same item.

The Analysis Determines What Else the Questionnaire Must Carry▼

A poverty threshold cannot be applied to personal income alone. It requires household size, and under most definitions geography as well, because the threshold differs between a single person in a small community and a family of four in a large city. An income-to-needs ratio requires the same companion items. If either appears in the analysis plan, the questionnaire must collect them, and the requirement becomes visible only when the analysis is worked through before the instrument is fielded. A coding scheme can be revised at any time. A companion item discovered to be missing after the field period has closed cannot be added.

Together these considerations imply that the coding scheme should be drafted alongside the questionnaire rather than after the data arrive. One practical test is to write the shell of the main results table before the instrument is finalised, with the rows and columns labelled as they will appear in the paper and the cells left empty. This forces every variable in the analysis to be traced back to an item on the questionnaire, and it exposes both the items that no analysis will use and the analyses that no item can support. Every coding decision that results, including reserved values, cut points, and derived variables, belongs in a codebook written and circulated with the questionnaire, so that the coder, the analyst, and any later user of the data read the same numbers in the same way.

A codebook fixes what each number means. The last step is to get those numbers into the computer without damaging them.

7.3 Data Entry and Management

Coding and data entry are separate jobs, and keeping them apart prevents errors that are very hard to find later. Responses should be coded directly on the paper questionnaire or data-capture form, using a distinctive ink colour so that the coder’s marks are easy to distinguish from the respondent’s answers, and only then entered. Combining coding and data entry into a single step increases the risk of errors. This is because the person doing both at once must read the respondent's answer, decide which code it deserves and type that code, all in one motion, and a slip at any of these points leaves no trace. When the codes are first written on the form, each decision is recorded on paper, so a later check can compare what was entered in the computer against what the coder wrote and find the mistake.

Computer data entry can be accomplished using general-purpose software such as spreadsheets or using specialised database managers. Each approach has advantages, and Table 3.3 compares the two on five features: ease of setup, validation rules, error prevention, the risk to data integrity and the transfer of data to statistical software.

Table 3.3. Comparison of spreadsheets and specialised database software for data entry across five features.

FeatureSpreadsheetsSpecialised Software
Ease of SetupEasyModerate
Validation RulesLimitedExtensive
Error PreventionLowHigh
Data Integrity RiskHigh (column sorting can destroy records)Low
Transfer to StatisticsManualBuilt-in

Spreadsheets are easy to set up, but Table 3.3 shows their weakness: limited validation rules, little error prevention and a high risk to data integrity. The most serious of these risks is worth remembering on its own.

Never Sort a Single Spreadsheet Column

Use spreadsheets with caution for data entry. Sorting individual columns in a spreadsheet can completely destroy the data by separating a respondent’s answers across different rows. In a data spreadsheet each row holds one respondent, so the only thing linking a person's age to their smoking status is that the two values sit on the same row. If one column, such as age, is sorted on its own, its values move to new rows while every other column stays put, and each age is now paired with somebody else's answers. Nothing on the screen looks wrong, which is why the damage is so dangerous. Specialised database managers or statistical software with built-in data-entry modules are generally safer choices for larger studies.

This section has set out strategies for maximising response rates and the principles for coding responses, deriving analysis variables and entering data. The key takeaways below summarise these points.

Key Takeaways

  • 7.1 A low response rate is more than an inconvenience: the people who do not respond often differ from those who do in their health, behaviour or attitudes, and when they do, your estimates are at risk of selection bias. (Strictly, the response rate is a proportion, the share of sampled people who complete the questionnaire, although everyone calls it a rate.) Strategies to raise response fall into two groups: design strategies, such as a clear purpose, a professional layout, thorough pre-testing, a short questionnaire and no unnecessary sensitive questions; and administrative strategies, such as small incentives, a stamped first-class return envelope, advance notice, a personalised cover letter and hand delivery. Following up non-responders with reminders and a fresh copy of the questionnaire is one of the most effective strategies, and small monetary incentives and recorded delivery also raise response substantially. When a scenario such as Worked Example 3.4 describes a low return, recommend changes from both groups, include follow-up of non-responders, and explain why it matters by linking low response to selection bias.
  • 7.2 A coding plan decides, before any data arrive, how every possible answer will be recorded as a number. Give missing answers a single unique code that can never be a real answer, such as -999, and never leave a field blank, because a blank cannot be told apart from an item the coder skipped; never code a missing answer as 0 either, because 0 can be a real value and would be averaged in with the others. Give “don’t know” and “prefer not to answer” (a refusal) separate codes of their own too, such as 7777 and 8888 for the income item in Table 3.2, because a refusal, a genuine “don’t know” and a skipped item mean different things, and refusals are often patterned rather than random. Use the same conventions throughout the questionnaire, such as 0 = no and 1 = yes for every yes/no item, and remember that reserved codes must be declared missing in the statistical software before analysis so that they are not treated as real values. When you write a coding plan, check that every response option, including the non-substantive ones, has a code, and that no reserved code falls inside the plausible range of the variable.
  • 7.3 Turning completed forms into a dataset is a separate job from coding them, and keeping the two steps apart prevents errors that are very hard to find later. Code the responses directly on the paper form, using a distinctive ink colour so the coder’s marks stand out from the respondent’s answers, and only then enter the data; combining coding and entry into one step increases the risk of mistakes. Spreadsheets are easy to set up but risky, because sorting a single column separates a respondent’s answers across different rows and silently destroys the dataset. Specialised database software, or statistical software with a data-entry module, is safer for all but the smallest studies, because it offers validation rules, stronger error prevention and built-in transfer to statistical analysis. If a question asks how to manage data entry, recommend coding on the form as a separate step and specialised software, and be ready to explain exactly how sorting one column can ruin a spreadsheet dataset.
Knowledge Check: this section

1. A health authority emails a 30-minute questionnaire on physical activity to 3,000 adults. After 4 weeks, 750 (25%) have replied, and most of the replies come from people who say they exercise regularly. What should the team do?

A sample of 750 sounds large, but size does not remove bias. When responders, mostly regular exercisers, differ from non-responders, more replies from the same group make the estimate more precise, not less biased.
Adding more detailed questions lengthens the questionnaire, which tends to lower response, and does nothing about who replied. The problem is selection, not the amount of information per reply.
Low response threatens validity when responders differ from non-responders, as here. Follow-up with reminders and a fresh copy is one of the most effective remedies, a shorter questionnaire helps, and the estimate should be read with caution.
Volunteers from a gym are likely to be even more active than the regular exercisers who already replied, so replacing non-responders with them would make the overestimate of physical activity worse, not better.
Correct answer: follow up the non-responders, shorten the questionnaire and treat the estimate with caution. The information needed is the response (750 ÷ 3,000 = 25%) and who replied: mostly regular exercisers, so responders appear to differ from non-responders on the very thing being measured. Low response threatens validity because, when non-responders differ systematically from responders, the estimate is biased (here, physical activity would be overstated); one of the most effective remedies is follow-up of non-responders with reminders and another copy, supported by design changes such as a shorter questionnaire. Analysing the 750 replies is tempting because the number sounds large, but more replies from the same unrepresentative group make the estimate more precise, not less biased, and volunteers from a gym would make the bias worse. (Section 4, slides “Non-response is a threat to validity” and “Follow up with non-responders”, 7.1.2 Strategies That Raise Response and Worked Example 3.4; Objective 7.1)

2. For the item “In the past 30 days, on how many days did you smoke a cigarette? ____ days” (possible answers 0 to 30), a research assistant proposes this coding plan: record the number of days; leave skipped items blank; code “Don’t know” as 0; code “Prefer not to answer” as 0. Which revised plan follows the lesson?

Here 0 is a real answer, meaning no days of smoking. Coding “Don’t know” and refusals as 0 would count them as non-smokers and pull the average down, and blanks cannot be told apart from items a coder missed.
This swaps the faults rather than fixing them: skipped items become 0 and are counted as non-smokers, while blanks for “Don’t know” and refusals cannot be told apart from fields left empty by mistake.
Moving every non-answer out of the 0 to 30 range is better than the original plan, but a single code loses the difference between a skipped item, a genuine “don’t know” and a refusal, which the lesson keeps apart.
Real answers keep the values 0 to 30, and each kind of non-substantive answer gets its own code outside that range, so nothing is miscounted as a non-smoker and no field is left blank. The codes are then declared missing before analysis.
Correct answer: days as 0 to 30, -999 for a skipped item, and separate codes for “don’t know” and refusals. The information needed is that 0 is a real answer to this item (a person who did not smoke), and that the plan uses blanks and 0 for answers that are not real values. Coding “Don’t know” and refusals as 0 would count them as non-smokers and pull the average down, and a blank cannot be told apart from an item the coder missed, so each non-substantive answer needs its own code outside the range 0 to 30, declared missing in the software before analysis. Coding all three as -999 is tempting and is better than the original plan, but it loses the difference between a skipped item, a genuine “don’t know” and a refusal, and refusals are often patterned in ways worth reporting. (Section 4, 7.2.1 Coding Principles, Missing Value Codes panel, and 7.2.2 From Response Options to Analysis Variables, Table 3.2 and the Non-Substantive Answers Need Codes of Their Own panel; Objective 7.2)

3. A student enters 400 completed questionnaires into a spreadsheet, one row per respondent. To find the oldest respondents, she selects only the age column and sorts it from highest to lowest. What has happened to the dataset?

Sorting one column does more than change the display: it reorders the ages while every other column stays where it was, so each row now pairs one person’s age with someone else’s answers.
Sorting one column moves almost every age, not just the oldest, so nearly every row now mixes data from different respondents. This silently destroys the dataset, which specialised data-entry software is designed to prevent.
Only the top rows look affected, but sorting a single column moves nearly every value in it, so almost every row now pairs one person’s age with another person’s answers, not only the rows for the oldest respondents.
Sorting does not change the type of the values; the ages remain numbers. The harm is that they have been separated from the rest of each respondent’s record, which retyping would not repair.
Correct answer: the ages have been separated from the rest of each record. The information needed is that the data are stored one row per respondent and that only the age column was sorted. Sorting one column reorders the ages but leaves every other column where it was, so each row now pairs one person’s age with another person’s answers, and the dataset is silently destroyed without any error message; specialised database or statistical software with data-entry modules enforces validation rules, prevents this and is safer for all but very small studies. It is tempting to think that only the top rows are affected, but nearly every age has moved, so nearly every row is now wrong. (Section 4, slide “Spreadsheets carry a specific risk”, 7.3 Data Entry and Management and Table 3.3; Objective 7.3)

4. A public-health unit mails a questionnaire on household food security to a random sample of 1,500 households. After four weeks, 390 completed questionnaires have come back. The team then mails a reminder and a fresh copy to every household that has not replied, and a further 345 are returned. What are the response rates before and after the follow-up?

The response rate uses the 1,500 households sampled as the denominator throughout and counts all completed questionnaires: 390 / 1,500 = 26.0% before follow-up and (390 + 345) / 1,500 = 49.0% after.
The first rate is right, but 345 / 1,500 counts only the second mailing and ignores the 390 replies already received. The response after follow-up is cumulative, (390 + 345) / 1,500 = 49.0%.
The first rate is right, but 345 / 1,110 is the share of followed-up households that replied, not the study’s response rate, which keeps all 1,500 sampled households in the denominator.
Dividing by 1,110, the non-responders after the first mailing, is a wrong-denominator error. The response rate divides completed questionnaires by all 1,500 households sampled.
Correct answer: 26.0% and 49.0%. The information needed is the number of households sampled (1,500, the denominator throughout) and the cumulative number of completed questionnaires (390, then 390 + 345 = 735). Before follow-up, 390 ÷ 1,500 = 0.26; after it, 735 ÷ 1,500 = 0.49, so following up the non-responders nearly doubled the response. Counting only the second mailing (23.0%) is tempting but ignores the first 390 replies; 31.1% is the share of followed-up households that replied, not the response rate of the study; and dividing by the 1,110 non-responders is a wrong-denominator error. (Section 4, 7.1.1 Why Low Response Threatens Validity and Worked Example 3.4; Objective 7.1)

5. In the same food-security study, suppose that food-insecure households are less likely than other households to return the questionnaire. What is the likely effect on the estimate of food insecurity?

Random sampling protects the sample drawn, not the subset that chooses to reply. If food-insecure households reply less often, the responders are no longer representative.
The direction is reversed. Households that return the questionnaire less often are under-represented among responders, not over-represented, so the proportion food insecure is understated.
Non-response does reduce precision, but when responders differ systematically from non-responders it also introduces bias, and more replies from the same unrepresentative group do not remove it.
Food-insecure households that reply less often are under-represented among responders, so the estimated proportion food insecure is too low. This is selection bias, which follow-up reduces but cannot remove while many still do not reply.
Correct answer: biased downward. The information needed is the direction of the difference between responders and non-responders: if food-insecure households reply less often, they are under-represented among the responders, so the proportion who are food insecure is underestimated. This is selection bias, and more replies from the same unrepresentative group make an estimate more precise, not less biased. Random sampling is tempting as a defence, but it protects the sample drawn, not the subset that chooses to reply. Following up non-responders reduces this risk but cannot remove it while many households still do not reply. (Section 4, 7.1.1 Why Low Response Threatens Validity and 7.1.2 Strategies That Raise Response; Objective 7.1)

6. The food-security questionnaire includes the item “In the past 12 months, did anyone in your household eat less than they felt they should because there was not enough money for food?” (Yes / No / Don’t know / Prefer not to answer). Every other yes/no item is coded 0 = no and 1 = yes. Which coding plan is best?

This plan breaks several principles: 1 = yes, 2 = no departs from the 0/1 convention used elsewhere, blanks cannot be told apart from coding errors, and giving “Don’t know” and refusals the same code loses the difference between them.
The yes/no codes and -999 are right, but coding “Don’t know” and refusals as 0 records them as “no”, so those households would be counted as food secure and the estimate would fall.
This plan keeps the 0/1 convention, gives skipped items, “don’t know” and refusals distinct codes outside the real answers, declares those codes missing before analysis and keeps coding on paper separate from data entry.
A single code for every non-answer loses the difference between a skipped item, a genuine “don’t know” and a refusal, and refusals are often patterned in ways worth reporting.
Correct answer: the plan with consistent yes/no codes and distinct reserved codes. The information needed is the convention used elsewhere (0 = no, 1 = yes) and the three kinds of non-substantive answer (skipped, don’t know and refusal) to keep apart. A good plan uses the same yes/no convention throughout, gives a skipped item, a “don’t know” and a refusal distinct codes outside the range of real answers, never uses a blank or 0 for a missing answer, declares the reserved codes missing before analysis, and keeps coding separate from data entry. Coding “Don’t know” and refusals as 0 is tempting because neither confirms food insecurity, but 0 means “no” here, so those households would be counted as food secure and the estimate would fall; a single code for all non-answers loses the difference between a skipped item, a genuine “don’t know” and a refusal, and refusals are often patterned. (Section 4, 7.2.1 Coding Principles and 7.3 Data Entry and Management; Objective 7.2)

✦ Pass the knowledge check with 100% to continue

Section 5

Final Review & Assessment

⏱ Estimated time: 20 minutes

Bringing It All Together

This lesson walked through the full lifecycle of a research instrument: planning, drafting items, refining wording and structure, pre-testing, fielding, and finally coding the responses for analysis. Each stage compounds: a poorly defined objective produces vague items, vague items produce unstable measurements, and unstable measurements produce associations that look like findings but are really just noise.

The threads pulled together here also matter beyond questionnaire design. Cognitive theory of survey response (Tourangeau & Yan, 2007), validation, and pilot testing show up again whenever epidemiologists rely on self-reported exposures or outcomes, and the total survey error framework (Groves & Lyberg, 2010) ties these threads together. As you head into the final assessment, treat the takeaways below as the practical companion to the logic of sampling: you now know who to recruit and how to ask them what you need to know.

Key Takeaways from this lesson

  1. You can now explain that a questionnaire is the instrument and a survey is the study that uses it, and you can describe how a questionnaire is planned: define the objectives, consult interest holders (for example with the Delphi technique), run focus groups of 6 to 12 people from the study population and review existing validated instruments. For the exam, remember that planning comes before a single item is written, so give the steps in order and say what each one contributes.
  2. You can compare in-person, telephone, mailed and internet administration on response rate, cost, interviewer bias, reach and the handling of sensitive questions, and choose a method for a described study. No method wins on every count, so work through who the respondents are, whether they can read the form, how sensitive the topic is and what the budget allows, and justify your choice on the features that matter most for that study.
  3. You can describe the four cognitive steps in answering a question (understanding, retrieval, judgement and communication) and say at which step a flawed item is likely to fail. When you diagnose an item, read it as a respondent would and stop at the first step where an honest person would struggle; a daily count over a month or a year, for example, fails at retrieval, and the remedy is a shorter reference period or a typical day in a recent week.
  4. You can choose an appropriate question format, open or closed, and among checklist, multiple-choice, rating and ranking questions, applying the design rule for each type. Ask for exact numbers such as age wherever you can, because continuous data can always be grouped later but a range can never be turned back into an exact value; income is the usual exception and is asked in non-overlapping ranges with “Don’t know” and “Prefer not to answer”.
  5. You can name the rule a draft item breaks (too long or full of jargon, not specific about the event or time frame, double-barrelled, leading, or with categories that overlap or leave gaps) and write an improved version. On the exam, name the specific rule rather than calling an item “unclear”, check that your rewrite removes that fault without introducing a new one, and test the boundary values to spot overlapping or incomplete categories.
  6. You can lay out a questionnaire with an introduction, easy opening items, a funnel from general to specific within each section, sensitive items near the end and pre-coded responses, and you can explain what expert review, field pre-testing, think-aloud interviews, test-retest reliability and validation against a gold standard each reveal. Keep the distinction clear: pre-testing checks that the questionnaire works for real respondents, while validation checks that it measures what it claims to measure.
  7. You can explain why low response threatens validity, recommend strategies to raise it, and write a coding plan that keeps coding separate from data entry. Include follow-up of non-responders in any list of strategies and link low response to selection bias, and in a coding plan give missing answers, “don’t know” and refusals distinct codes, never a blank and never 0.

Final Knowledge Assessment

This assessment covers all material from this lesson. You must score 100% to complete the lesson. Review the feedback for any incorrect answers and try again.

Final Assessment: Questionnaire Design (17 Questions)

1. A team developing a questionnaire on heat-related illness among older adults has two planning needs: (1) agreement among public-health officials, family physicians and housing advocates on the ten outcomes the questionnaire must measure, and (2) the words that older adults themselves use to describe symptoms of heat illness. Which pairing of planning tools fits these needs?

The Delphi technique is a structured process for building consensus among diverse interest holders, which fits need (1), and a moderated focus group of 6 to 12 people from the target population surfaces their own terms, which fits need (2).
The pairing is reversed. A focus group explores the attitudes and language of the study population, while the Delphi technique gathers agreement among interest holders rather than exploring how people describe symptoms.
A test-retest study and expert review both check items that have already been written. Neither builds consensus among interest holders or captures the words older adults use, and both belong to pre-testing, not planning.
A field pre-test checks a draft that already exists by showing which items real respondents struggle with. It cannot produce agreement on what to measure or the respondents’ own words before any items are drafted.
Correct answer: the Delphi technique for (1) and a focus group for (2). The information needed is the purpose of each task: (1) asks for consensus among diverse interest holders, and (2) asks for the language and concerns of the target population. The Delphi technique is a structured process for building consensus among interest holders, while a focus group of 6 to 12 people from the study population, guided by a moderator, surfaces attitudes and locally meaningful terminology, so each tool matches one need. The reversed pairing is tempting because both tools involve consultation, but a Delphi process gathers agreement rather than exploring how people describe their experience, and a field pre-test or test-retest study checks items that have already been written. (Section 1, slides “Four planning steps” and “Focus groups in questionnaire development”, 1.2 Planning a Questionnaire and 1.2.1 Focus Groups; Objective 1.2)

2. In a pilot study of vaping in two similar secondary schools, students in one school were asked the items in person by an interviewer, and 8% reported vaping in the past 30 days. Students in the other school completed the same items privately on a web form, and 19% reported vaping. Using Table 3.1, what is the most likely explanation for the difference?

This reverses Table 3.1: in-person interviews give the highest response of the four methods, not the lowest, so poor participation does not explain the lower figure.
The lesson gives no basis for thinking that web forms cause exaggeration. It says the opposite: answers given in private tend to be more candid, so the higher figure is more likely to be close to the truth.
With the items unchanged, the mode is the main difference. Table 3.1 notes that answers to sensitive questions given to an interviewer may be shaded toward what the interviewer seems to expect, while private answers tend to be more candid.
Both schools received the same items, so the wording cannot explain the difference. A double-barrelled item would be faulty in either mode; what changed was whether an interviewer was present.
Correct answer: the answers given to the interviewer were probably shaded, and the private web answers are probably more candid. The information needed is that the items were the same but the mode differed (an interviewer versus private self-completion) and that vaping by students is a sensitive behaviour. Table 3.1 notes that answers to sensitive questions given to an interviewer may be shaded toward what the interviewer seems to expect, whereas self-completed answers given in private tend to be more candid, so under-reporting in the interviews is the likeliest explanation for 8% versus 19%. The first option is tempting, but in-person interviews give the highest response of the four methods, not the lowest, and nothing in the stem suggests that web respondents exaggerate. (Section 1, 2.1 Methods of Administration and Table 3.1; Objective 2.1)

3. A team will study diarrheal illness among farm-worker families living in seasonal camps. Many adults have limited reading ability in English, few have internet access or a reliable telephone, and the team can travel to the camps. Which administration method fits best?

Mailed questionnaires are cheap and free of interviewer bias, but they require literacy, which many of these adults lack in English. A form respondents cannot read yields low response and poor data.
Only in-person interviews meet these conditions: they need no reading ability, internet or telephone, allow show-cards and other visual aids and give the highest response. Their main weakness, interviewer bias, is limited by training interviewers.
An internet questionnaire is low in cost and wide in reach, but it requires both literacy and internet access, and few of these families have access. Reach is irrelevant if respondents cannot use the method.
Telephone interviews need no reading ability and cost less than in-person interviews, which is true, but they need a reliable telephone, which few of these families have, so the method cannot reach them.
Correct answer: in-person interviews by trained interviewers. The information needed is the respondents’ literacy, their access to the internet and telephones, and whether the team can reach them in person. Table 3.1 shows that mailed and internet questionnaires require literacy (and the web also requires access), and telephone interviews need a telephone, so only in-person interviews can reach these families; they need no reading ability, allow visual aids and give the highest response, and their main weakness, interviewer bias, is reduced by careful training of interviewers. Mailed questionnaires are tempting because they are cheap and free of interviewer bias, but a form that many respondents cannot read will produce low response and poor data. (Section 1, slide “Four methods, each with trade-offs”, 2.1 Methods of Administration and Table 3.1; Objective 2.1)

4. A questionnaire for parents asks: “Has your child had an episode of otitis media in the past 6 months?” (Yes / No / Not sure). At which cognitive step will many parents struggle, and what is the fix?

Six months is a moderate period for a memorable event such as an ear infection, so retrieval is not the main problem. Shortening the period to 7 days would miss most infections and leave the medical term in place.
The item asks about a factual event, not an opinion, so it does not depend on a subjective judgement. A rating scale would not help a parent who does not know what “otitis media” means.
Yes / No / Not sure can record whether an infection occurred, so communication is not where the item fails. Adding categories would not help a parent who cannot understand the question.
Understanding, the first cognitive step, requires clear, non-technical language. Many parents will not know “otitis media”, so they cannot answer however well they remember; plain words such as “an ear infection” fix the item.
Correct answer: understanding. The information needed is the wording: “otitis media” is a medical term, and the first step, understanding, requires clear, non-technical language. A parent who does not know the term cannot answer accurately however well they remember, so the fix is plain words, for example “Has your child had an ear infection in the past 6 months?” Retrieval is tempting because six months sounds long, but an ear infection is a memorable event over a moderate period, and shortening the period to 7 days would miss most infections without removing the jargon. (Section 2, 3.1 The Four Cognitive Steps of Answering a Question, Figure 3.1 and Worked Example 3.1; Objective 3.1)

5. A draft item reads: “Do you drink too much coffee? (Yes / No)”. At which cognitive step is it most likely to fail, and what is the fix?

This item asks respondents to recall no amount and names no period, so retrieval is not where it fails, and shortening a period it does not have would not help. The trouble is the subjective phrase “too much”.
“Coffee” is an everyday word that respondents will understand, so the item does not fail at understanding. The unclear part is “too much”, which each respondent must define for themselves.
“Too much” has no fixed meaning, so each respondent makes a private, subjective decision, a judgement failure that makes answers incomparable. Asking for a concrete amount over a defined period, such as cups on a typical day, removes the judgement.
Yes / No options can record a yes-or-no verdict, so communication is not the first failure. The problem arises earlier, when respondents must decide what “too much” means.
Correct answer: judgement. The information needed is the phrase “too much”, which has no fixed meaning: one respondent may think two cups a day is too much, and another may think six is fine. The item asks for a subjective decision, so it fails at the judgement step and answers cannot be compared across respondents; the fix is to make it less subjective by asking for a concrete amount, such as “On a typical day in the past 7 days, how many cups of coffee did you drink?” Retrieval is tempting because amounts of coffee can be hard to recall, but this item asks the respondent to recall no amount and no period at all, and Yes / No options can record a yes-or-no verdict, so communication is not the first failure either. (Section 2, 3.1 The Four Cognitive Steps of Answering a Question, Figure 3.1 and Worked Example 3.1; Objective 3.1)

6. A patient-satisfaction item reads: “How satisfied were you with your visit? ☐ Satisfied ☐ Dissatisfied”. The team plans to report the mean satisfaction score. Which revision follows the lesson’s rules for rating questions?

These are the lesson’s rules for rating scales: at least five points to avoid losing information, a deliberate decision about a neutral midpoint, a “Don’t know / Not applicable” option to separate non-response from neutrality, and a mean only for roughly equally spaced points.
A two-point item is simple, but it causes serious loss of information, which is why the lesson asks for at least 5 to 7 points. A mean of 0s and 1s cannot show degrees of satisfaction.
Adding “Don’t know” is one of the rules, but not the main or only one. Keeping two points leaves the serious loss of information, and a mean still needs at least five roughly equally spaced points.
This reverses what ranking does. Ranks record only order, so the distance between them is unknown, and a single satisfaction item offers nothing to put in order.
Correct answer: at least five labelled points, a decision about the midpoint, a “Don’t know / Not applicable” option, and a mean only for equally spaced points. The information needed is the scale as drafted (two points) and the planned analysis (a mean). Rating scales need at least 5 to 7 points to avoid serious loss of information, a deliberate choice between a neutral midpoint and a forced choice, and a “Don’t know” or “Not applicable” option that separates non-response from true neutrality; means are reasonable only with at least five roughly equally spaced points. Adding “Don’t know” alone is tempting because it is one of the rules, but it leaves the information loss of a two-point scale, and ranking would make matters worse, because the intervals between ranks are unknown. (Section 2, slide “Design rules that protect your data” and 4.2 The Four Types of Closed Question, Rating and Ranking tabs; Objective 4.2)

7. A draft item reads: “How long have you lived at your current address? ☐ Less than 1 year ☐ 1 to 5 years ☐ 5 to 10 years ☐ More than 10 years”. Which respondent cannot be placed in exactly one category, and what is the simplest repair?

A person who moved in last month fits “Less than 1 year”, so no category is missing at the lower end and a “None” category is not needed.
A person at exactly 5 years fits both “1 to 5 years” and “5 to 10 years”, so the categories are not mutually exclusive. Asking for the exact number of years removes the overlap and can be grouped later.
A person who has lived there 20 years already fits “More than 10 years”, so the top of the scale is covered and adding a “20 years or more” box would create a new overlap.
Checking the boundary values shows a fault: exactly 5 years fits both “1 to 5 years” and “5 to 10 years”, so the categories are not mutually exclusive and the item needs repair.
Correct answer: someone who has lived there exactly 5 years. The information needed is the boundary values, so check 1, 5 and 10 years and the two ends of the scale. A person at exactly 5 years fits both “1 to 5” and “5 to 10”, so the categories are not mutually exclusive, while every other length of residence, from a few weeks to more than 10 years, fits one category. The simplest repair is to ask for the exact number of years, which can be grouped later. The newcomer option is tempting because gaps at the ends are a common fault, but “Less than 1 year” already covers someone who has just moved in. (Section 2, slide “Four types, four sets of rules” and 5.1 Mutually Exclusive and Jointly Exhaustive Categories, Worked Example 3.2; Objective 5.1)

8. Which of these draft items best follows the lesson’s guidance on collecting numbers?

Banding a dentist-visit count into three broad groups throws away detail that an exact number would keep. Visits are not sensitive, so the lesson’s advice is to ask for the exact count, which can be grouped later.
This item does ask for an exact number, but earnings are income, the usual exception: an exact amount invites refusals and guesses, so ranges with “Don’t know” and “Prefer not to answer” work better.
Two age groups discard nearly all the detail in age, which is rarely sensitive. An exact age can be grouped later into any categories, whereas these two groups can never be turned back into exact ages.
Household size is not sensitive, and this item asks for it as an exact count with its unit stated, which follows the guidance to capture numbers as exact values that can be grouped later.
Correct answer: the household-size item. The information needed is the guidance itself: capture a number as an exact value wherever respondents will supply it, because exact values can be grouped later but ranges cannot be turned back into exact values, and make an exception for sensitive quantities such as income. Household size is not sensitive and is asked as an exact count with its unit, so it follows the guidance, whereas the dentist and age items throw away detail by forcing answers into a few broad groups. The earnings item is tempting because it also asks for an exact number, but earnings are income, the usual exception, and an exact amount invites refusals and guesses; ranges with “Don’t know” and “Prefer not to answer” work better. (Section 2, slide “Capture continuous data wherever possible” and 4.1 Open Questions, Fill-in-the-Blank Questions, Capturing Sensitive Data and Specifying Units and Scales panels; Objective 4.1)

9. A draft item reads: “Given the well-known dangers of second-hand smoke, do you support a ban on smoking in cars carrying children?” Which rule of question design does it break?

The opening phrase, “Given the well-known dangers of second-hand smoke”, reminds respondents of one side of the argument before they answer, which pushes them toward “yes”: the item is leading.
Two topics are mentioned, but the item asks only one question, whether respondents support the ban. The premise about second-hand smoke is there to persuade, not to be answered, so the fault is leading wording.
Rule 2 concerns behaviours and events that need a defined period; an opinion about a proposed ban does not need a time frame. The fault is the persuasive premise, not a missing period.
Mutually exclusive categories are a rule for response options. This fault lies in the wording of the question, its persuasive premise, and the lesson’s neutral version would still use balanced response options.
Correct answer: leading. The information needed is the opening phrase, “Given the well-known dangers of second-hand smoke”, which reminds respondents of one side of the argument before they answer and so pushes them toward “yes”. A neutral version drops the persuasive premise and names both directions, for example “Do you support or oppose a ban on smoking in cars carrying children?” with balanced response options. Double-barrelled is tempting because two topics are mentioned, but the item asks only one question (support for the ban); the premise is there to persuade, not to be answered. (Section 3, slide “Four rules that protect validity” and 5.2 Four Rules for Wording Questions, Rule 4 panel; Objective 5.2)

10. A draft item reads: “Have you recently experienced any adverse events following immunisation?” It breaks two wording rules: it uses jargon, and it is not specific. Which rewrite removes both faults?

Adding the abbreviation AEFI makes the item harder, not easier, to understand, and “recently” is still vague, so both faults remain.
Plain words such as “side effects” and “a vaccine” remove the jargon, but the item still has no time frame, so it remains not specific and respondents will each choose their own period.
This rewrite replaces the jargon with plain symptoms and names a short, recallable period, the 7 days after the most recent vaccination; its “Other” and “None of these” options also make the list jointly exhaustive.
Naming “the past month” fixes the vague time frame, but the jargon “adverse events following immunisation” remains, so the item still fails the rule on clear, non-technical language.
Correct answer: the checklist that names a 7-day period and plain symptoms. The information needed is the two faults in the draft: “adverse events following immunisation” is jargon (Rule 1), and “recently” is vague, with no defined period (Rule 2). The correct rewrite replaces the jargon with plain words for the events and names a short, recallable time frame, and its “Other” and “None of these” options make the list jointly exhaustive. The plain-language version and the “past month” version are tempting because each fixes one fault, but the first still has no time frame and the second keeps the jargon; adding the abbreviation AEFI makes the item harder, not easier, to understand. (Section 3, 5.2 Four Rules for Wording Questions, Rules 1 and 2 panels, and Worked Example 3.3; Objective 5.2)

11. A study asks: “How many migraines did you have in the past 30 days? ____”. In pre-testing, one respondent counts a headache that lasted three days as one migraine, while another counts it as three. Which rule of question design does the item break, and what is the fix?

The word “migraine” names the event; it does not suggest which answer is expected. The pre-test problem is that respondents count episodes differently, which a change of word would not solve.
Rule 2 asks the designer to name the time frame and define the event. The 30-day period is clear, but what counts as one migraine is not, so a rule for when a new migraine begins makes answers comparable.
The item asks one thing, the number of migraines; “days” describes the period, not a second behaviour. Splitting it would not tell respondents how to count a three-day headache.
Ranges would not define the event, so respondents would still count the same headache differently, and the ranges offered overlap at 2 and at 4. A fill-in count is the better format once the event is defined.
Correct answer: not specific, because the event is not defined. The information needed is the pre-test finding: the time frame (30 days) is clear, but respondents count episodes differently, so the item does not say what one migraine is. Rule 2 asks the designer to name the time frame and clearly define the event, as in the lesson’s example of counting a new episode of diarrhea only after at least 2 days of normal bowel movements; a similar rule here (for example, a new migraine starts only after a full day without headache) makes answers comparable. Replacing the blank with ranges is tempting as a tidy fix, but it does not define the event, and the ranges offered overlap at 2 and at 4. (Section 3, 5.2 Four Rules for Wording Questions, Rule 2 panel, and Worked Example 3.3; Objective 5.2)

12. When the same 50 adults complete a food frequency questionnaire twice, two weeks apart, their answers agree very closely. A colleague concludes: “So the questionnaire is valid: it measures true intake.” What is the best response?

Consistent answers show repeatability, not truth. Respondents who always under-report snacks will under-report them both times, so close agreement can sit around the wrong value.
The lesson describes a sensible test-retest interval as long enough for people to forget their answers but short enough that intake has not changed; two weeks fits, and six months would let true intake change.
Expert review checks coverage and logic, not whether answers match true intake. The lesson’s route to validity is comparison with a gold standard or an established instrument.
The questionnaire was compared only with itself, which assesses repeatability (test-retest reliability). Validity asks whether it measures true intake, which needs comparison with a gold standard such as weighed food records.
Correct answer: the result shows repeatability, not validity. The information needed is what was compared: the questionnaire with itself on two occasions, not with a measure of actual intake. Test-retest administration assesses repeatability, provided the questionnaire is unchanged and the interval is long enough that people do not remember their answers but short enough that intake has not changed (two weeks fits); validation asks whether the questionnaire measures what it claims, which needs a comparison with a gold standard such as directly weighed food records. The colleague’s conclusion is tempting because consistent answers feel trustworthy, but respondents who always under-report snacks will under-report them both times, giving close agreement around the wrong value. (Section 3, 6.2.1 Pre-Testing Methods, Test–Retest tab, and 6.2.2 Validation and the Cost of Unreliable Measurement; Objective 6.2)

13. Of 1,000 people who opened an online health survey, 300 (30%) closed it on the first screen, which had no introduction and asked for their exact household income. Which change follows the lesson?

This applies subsection 6.1: an introduction explaining purpose, use of the data, confidentiality and time, then easy, non-threatening items, with the sensitive income item near the end and asked in ranges to reduce refusals.
Making the item compulsory stops skipping, but people unwilling to give their income would simply close the survey, so drop-out would rise. It also keeps the sensitive item at the very start.
Removing introductory text is the opposite of the advice: the lesson says to open with an introduction covering purpose, use of the data, confidentiality and time needed, and the sensitive item would still come first.
Moving income later helps, but asking for an exact amount ignores the exception for sensitive quantities: ranges with “Don’t know” and “Prefer not to answer” reduce refusals. Exact numbers are preferred, but not always.
Correct answer: add an introduction and easy opening items, move income near the end and ask it in ranges. The information needed is where people stopped (300 of 1,000, or 30%, on the very first screen) and what that screen held: a sensitive item, asked as an exact amount, with no introduction. Subsection 6.1 says to open with an introduction and easy, non-threatening questions; sensitive items belong near the end, after rapport has built, so that someone who stops there has already answered everything else; and income is the usual exception to exact numbers, because ranges reduce refusals. Making the item compulsory is tempting because it stops people skipping it, but people who will not give their income would simply close the survey, so it would raise rather than lower the drop-out. (Section 3, 6.1 Structure and Layout of a Questionnaire; Section 2, 4.1 Open Questions, Capturing Sensitive Data panel; Objective 6.1)

14. To save time, a research assistant reads each paper questionnaire and types a code for each answer straight into the database, deciding on the codes as she goes. What is the main problem with this procedure?

Using -999 for missing answers is a correct rule, but it does not address the real problem: codes are being invented on the fly and entered in the same step, with no coding plan to check them against.
The lesson does not require online completion. Paper forms are acceptable; what matters is planning the codes in advance, coding on the form, and entering the data as a separate step.
Combining coding and entry makes errors very hard to detect. The lesson’s procedure is to plan the codes before fielding, code on the form in a distinctive ink colour, and then enter the codes as a separate step.
A spreadsheet would not solve the problem of combined coding and entry, and it adds a risk: sorting one column can destroy records. Specialised data-entry software, which enforces validation rules, is the safer choice.
Correct answer: coding and data entry have been combined. The information needed is that the assistant both chooses the codes and enters them in a single pass, with no coding plan drafted beforehand. The lesson says to plan the coding before the questionnaire is fielded, to code directly on the form in a distinctive ink colour so that the coder’s marks stand apart from the respondent’s answers, and to enter the data as a separate step, because combining the two produces errors that are very hard to detect afterwards. Using -999 for missing answers is tempting because it is a correct rule, but it does not solve the problem of codes invented on the fly with no record to check them against, and a spreadsheet would add the risk of destroying records by sorting. (Section 4, slide “Three principles that protect data integrity”, 7.3 Data Entry and Management and the end of 7.2.2 From Response Options to Analysis Variables; Objective 7.3)

15. A mailed questionnaire on chronic pain is sent to 1,500 adults, and 450 return it. Which statement gives the correct response rate and the correct interpretation?

Dividing by the 1,050 non-responders is a wrong-denominator error; the response rate divides returns by all 1,500 sent. Low response matters mainly because responders may differ from non-responders, not because of sample size.
450 / 1,500 = 30%. With most adults not replying, the main concern is that responders differ from non-responders, for example people in pain being keener to reply, which would bias the estimate; following up non-responders is one of the most effective remedies.
The rate is right, but sample size does not protect against bias. If responders differ systematically from non-responders, 450 replies give a precise estimate of the wrong value.
1,050 / 1,500 = 70% is the proportion who did not respond, not the response rate. The response rate is 30%, and a low response, if anything, raises the risk of selection bias.
Correct answer: 30%, with a risk of selection bias. The information needed is the number sent (1,500, the denominator) and the number returned (450, the numerator): 450 ÷ 1,500 = 0.30, or 30%, which is a proportion despite the name “response rate”. With 70% not responding, the concern is not the sample size but whether responders differ from non-responders; if people with chronic pain are keener to reply, the prevalence of pain will be overestimated, and following up non-responders is one of the most effective remedies. Dividing by the 1,050 non-responders (43%) is a wrong-denominator error, and 70% is the proportion who did not respond. (Section 4, 7.1.1 Why Low Response Threatens Validity, 7.1.2 Strategies That Raise Response and slide “Follow up with non-responders”; Objective 7.1)

16. A city health department plans a questionnaire to estimate how many residents aged 18 to 29 drove a motor vehicle within 2 hours of using cannabis in the past 30 days. It wants about 600 completed questionnaires; a similar survey suggests that about 30% of those invited will respond, and almost all young adults in the city use the internet. Which plan is best supported?

Multiplying 600 by 0.30 gives the completed questionnaires expected from 600 invitations, not the invitations needed. Inviting 180 would yield only about 54 replies; invitations needed = completed wanted ÷ expected response.
Dividing by 0.70 uses the expected non-response instead of the expected response. With 30% responding, 857 invitations would yield about 257 replies, far short of 600.
The arithmetic is right, but in-person interviews are the most expensive method, and an interviewer may shade answers about a sensitive behaviour toward what the interviewer seems to expect, which reverses Table 3.1.
Invitations needed = completed questionnaires wanted ÷ expected response = 600 / 0.30 = 2,000. A private web questionnaire suits young adults who almost all use the internet and a sensitive topic, and reminders raise its lower response.
Correct answer: about 2,000 invitations to a web questionnaire. The information needed is the target number of completed questionnaires (600), the expected response (30%), and the sensitivity of the topic. Invitations needed = completed questionnaires wanted ÷ expected response = 600 ÷ 0.30 = 2,000. Almost all young adults use the internet, a web questionnaire is cheap and wide in reach, and answers given in private about a sensitive behaviour tend to be more candid than answers given to an interviewer; its lower response can be offset with reminders. Multiplying by the response (180) or dividing by the non-response (857) are tempting arithmetic slips, and in-person interviews are the most expensive method and may shade sensitive answers toward what the interviewer seems to expect. (Section 1, 2.1 Methods of Administration and Table 3.1; Section 4, 7.1.1 Why Low Response Threatens Validity and 7.1.2 Strategies That Raise Response; Objective 2.1)

17. A first draft of the same questionnaire asks: “How often do you drive after using cannabis or drinking alcohol? ☐ Never ☐ Sometimes ☐ Often”. Which statement correctly diagnoses and repairs the item?

This diagnosis names every fault: two behaviours share one answer, there is no reference period, “after” has no limit, and “sometimes” and “often” are vague. One behaviour, a defined time window and a count of days repair it.
The item is not leading, since it does not push respondents toward an answer, and the yes/no rewrite keeps both behaviours together, still has no time frame and loses the frequency the study needs.
Splitting the item fixes the double barrel, but each new item still has no reference period, “after” is still undefined, and “sometimes” and “often” remain vague, so the item is still not specific.
Adding “In the past 30 days” fixes the time frame, but cannabis and alcohol still share one answer and the vague options remain, so the item is still double-barrelled and imprecise.
Correct answer: double-barrelled and not specific, repaired with one behaviour, a reference period and a count. The information needed is every fault in the item: two behaviours share one answer, there is no time frame, “after” has no limit, and “sometimes” and “often” mean different things to different respondents. An exact number of days is better than categories, because a count can be grouped later in any way the analysis needs. Splitting the item, or adding a time frame, is tempting because each fixes a real fault, but each leaves the other fault in place, and a yes/no item loses the frequency the study needs. (Section 3, 5.2 Four Rules for Wording Questions and Worked Example 3.3; Section 2, 4.1 Open Questions; Objective 5.2)

✦ Pass every section knowledge check with 100% before submitting