Questionnaire
Design
Fundamental Epidemiological Concepts and Approaches
Learning objectives for this lesson:
- Distinguish a questionnaire from a survey, and describe the steps in planning a questionnaire: defining objectives, consulting interest holders (for example with the Delphi technique), running focus groups and reviewing existing validated instruments.
- Compare in-person, telephone, mailed and internet administration on response rate, cost, interviewer bias, reach and the handling of sensitive questions, and choose a method for a described study.
- Describe the four cognitive steps in answering a question (understanding, retrieval, judgement and communication), and identify the step at which a flawed item is likely to fail.
- Choose an appropriate question format (open or closed; checklist, multiple-choice, rating or ranking), and explain why exact numbers such as age are preferred to ranges, with income as the usual exception.
- Identify the rule of question design that a draft item breaks (keep it short and simple, be specific, avoid double-barrelled and leading wording, and use response categories that are mutually exclusive and jointly exhaustive), and write an improved version.
- Describe how to structure and lay out a questionnaire (an introduction, easy opening items, the funnel approach, sensitive items near the end and pre-coded responses), and explain what expert review, field pre-testing, think-aloud interviews, test-retest reliability and validation against a gold standard can each reveal.
- Explain why low response threatens validity, recommend strategies to raise response (including follow-up of non-responders), and write a coding plan that gives missing answers, “don’t know” and refusals distinct codes and keeps coding separate from data entry.
This course was developed by Dr. Kiffer G. Card based on Dohoo, I. R., Martin, S. W., & Stryhn, H. (2012). Methods in Epidemiologic Research. VER Inc.
Glossary: Key Terms, People & Concepts
📚 Reference page, available throughout the lesson
This glossary collects the key concepts, people, and ideas you will meet in this lesson. Use it as a reference while you work through the material, or as a review before assessments. Type in the search box to filter entries.
Planning a Questionnaire
⏱ Estimated reading time: 12 minutes
Overview
The analysis of a survey depends on its sampling design. This lesson takes up the very next decision: once you have decided who to sample, you have to decide what to ask them. The questionnaire is where measurement actually happens, where exposures, outcomes and confounders enter the dataset as raw responses. Done well, it preserves the validity work begun at the sampling stage. Done badly, no amount of clever sampling or statistical modelling later can save the study. The lesson moves from planning a questionnaire, through the choice of question formats and the wording, structure and pre-testing of questions, to response rates and data coding. This first section explains what a questionnaire is and how it differs from a survey, how a questionnaire is planned, the two broad types of questionnaire, and the four ways of administering one.
Learning Objectives
- 1.1 Define questionnaire and survey, and distinguish between them.
- 1.2 Explain the key steps in planning a questionnaire, including consultation with interest holders (the Delphi technique), focus groups and the review of existing instruments.
- 1.3 Compare qualitative and quantitative questionnaire types, and say when each is used.
- 2.1 Compare the four administration methods on response rate, cost, interviewer bias, reach and the handling of sensitive questions.
1.1 Questionnaires and Surveys
Questionnaires are among the most commonly used tools for collecting data in epidemiological research. The terms questionnaire and survey are often used interchangeably in everyday speech, but in research they have distinct meanings. Click each of the two cards below to read the definition of each term.
Instrument or Study?
The questionnaire is the instrument; the survey is the study that uses it. Decisions about the survey, such as how the sample is drawn, are separate from decisions about the questionnaire, such as how an item is worded, and both affect how the answers can be interpreted.
The distinction matters because the quality of the instrument limits the quality of the study. As a primary means of data collection, questionnaires play a significant role in the quality of epidemiological research, yet less attention is often paid to the data-collection process than to the analytical methods used later. A well-designed questionnaire is essential for producing reliable, valid data, which Groves & Lyberg (2010) describe as minimising total survey error. Total survey error is the combined amount by which a survey estimate can miss the true population value. By this we mean that error does not come from a single source: it builds up from several, and each one can push the final estimate away from the truth. Sampling error arises because only some members of the population are asked; coverage error arises when the list used to draw the sample leaves some people out; non-response error arises when the people who do not answer differ from those who do; and measurement error arises when the answers people give do not match their true values. Because these errors add together, a survey with a perfect sample can still produce a misleading estimate if the questions themselves are poorly worded. The questionnaire governs measurement error directly, since every confusing or leading item can distort an answer. In addition, through its length and design it also affects non-response, because a long or off-putting form leads more people to give up, a theme taken up in Section 4.
Because the questionnaire carries so much of the weight, it needs its own planning before a single item is written.
1.2 Planning a Questionnaire
Effective questionnaires require careful planning. The very first step is to establish the objectives and information requirements of the study, and this involves consultation with subject experts, with the ultimate users of the information and, ideally, with members of the population to be surveyed. Planning then proceeds through four steps: define the objectives, consult interest holders, review existing instruments and engage the target population. The order matters, because each step shapes what the items must cover and how they should be worded, so item writing comes only after all four. Expand each step below to see what it involves and who takes part.
Clearly identify what information the study needs to collect. What are the key research questions? What variables need to be measured? Consulting with subject experts and interest holders at this stage helps ensure all important areas are covered.
Engage with subject experts and the end users of the data. If the data will be used by policymakers, they should be part of the planning process, because a questionnaire that omits the information they need cannot inform their decisions. When many interest holders are involved, they will often disagree about what the questionnaire should cover. A structured process like the Delphi technique can help build consensus among these diverse interest holders. By this we mean a method in which a panel of experts answers a series of questionnaires in rounds, usually without meeting face to face. After each round, the organiser summarises the panel's responses and shares the summary, and each member can then revise their views in light of what the others said. Because the rounds are anonymous, no single senior or outspoken person can dominate, and the repeated feedback tends to move the panel toward agreement on which topics and variables matter most.
Search for previously published questionnaires on the same topic. These are especially valuable if they have undergone formal validity assessments. By this we mean that other researchers have already tested whether the questions measure what they claim to measure, for example by comparing answers about a condition with clinical records. Because that testing has already been done, building on validated instruments can save time and improve the quality of your data collection, and it also lets your results be compared with those of earlier studies that used the same questions.
Members of the population to be surveyed should also be consulted during the planning phase. Their input can reveal important issues, concerns, and language preferences that researchers might otherwise overlook.
Two of these steps have well-developed methods of their own: focus groups, which serve the consultation in steps 2 and 4, and the re-use of validated instruments in step 3.
1.2.1 Focus Groups
Focus groups are a valuable tool during questionnaire planning because they consult members of the study population and the end users of the information directly. A focus group consists of 6 to 12 people drawn from the intended study population and/or the end users of the information. An independent moderator guides the discussion to ensure that it stays on track and that no single individual dominates, and it is often helpful to record the discussion for later review. Focus groups provide insight into attitudes, opinions, concerns and experiences. For the questionnaire designer, they help clarify objectives, identify data requirements, define salient terminology and concepts, and surface research issues that need to be addressed. A focus group may show, for instance, that respondents use an everyday word in a different sense from the researchers, in which case the questionnaire must define the term in plain words.
1.2.2 Re-Using Validated Instruments
Step 3 can save more time than any other. Before drafting brand-new items, check whether validated questions already exist in a major Canadian instrument. Re-using items shortens the timeline, because the work of drafting and testing the wording has already been done. In addition, it lets you benchmark your sample against national or provincial norms. By this we mean that when your question is worded exactly as it is in a national survey, your answers can be set directly beside the national figures. For example, if your community survey asks about food insecurity using the same items as the national survey, you can say whether food insecurity in your community is higher or lower than in the province as a whole. If you had written your own question instead, any difference might simply reflect the change in wording rather than a real difference between populations. This immediate population comparison is a major practical advantage.
Four sources are especially useful. Statistics Canada publishes the full Canadian Community Health Survey (CCHS) questionnaire, with derived variables and skip patterns, for every cycle back to 2001; its modules cover chronic conditions, mental health (e.g., PHQ-9, K6), substance use, food security, access to care, and many "rapid response" topical content blocks. The CHMS questionnaire is structured around the direct-measurement clinic visit and is useful for diet recall, physical activity (with accelerometer protocols) and biomarker collection items. The PHAC mental-health and chronic-disease indicator frameworks, such as the Positive Mental Health Surveillance Indicator Framework, specify the question wording behind national surveillance indicators. Finally, BCCDC and provincial community surveys, for example the BC Adolescent Health Survey (McCreary Centre Society), the BC SPEAK community surveys and BCCDC-led behavioural surveillance instruments on STBBI and substance use, supply items for provincial comparisons.
Planning settles what the study needs to know. The next decision is what kind of questionnaire will collect it.
1.3 Qualitative and Quantitative Questionnaires
Questionnaires can be broadly classified as qualitative or quantitative. The two types differ in the kind of questions they use and in the stage of research they serve, and structured, quantitative questionnaires are the main focus of epidemiological research and of the rest of this lesson. Open each tab to compare the two types.
Qualitative (Explorative) Questionnaires
Qualitative questionnaires are sometimes called explorative questionnaires. They consist primarily of open questions that allow participants to express their views and thoughts freely. These are used during the hypothesis-generation phase of research when researchers need to identify as many issues as possible related to the topic. They are often administered through interviews and may be audio or video recorded.
Quantitative (Structured) Questionnaires
Quantitative, or structured, questionnaires are designed to capture specific information about study subjects and their environment using mostly closed questions. The responses can be easily coded and analysed statistically. These are the primary focus of questionnaire design in epidemiological research.
The two types often work in sequence. The focus groups described in 1.2.1 belong to a qualitative phase that finds out which issues matter and how people talk about them, and the quantitative questionnaire that follows measures how common those issues are. To decide between the types for a particular study, look at its aim: exploring what matters points to a qualitative questionnaire, while measuring how common something is or comparing groups points to a quantitative one.
Once the type of questionnaire is settled, the way it is administered affects both the response rate and the quality of the data.
2.1 Methods of Administration
How a questionnaire is administered can have a substantial impact on both the response rate and data quality. There are four methods (in-person interviews, telephone interviews, mailed questionnaires and internet questionnaires), and none is best on every count. Click each card below to see the advantages and disadvantages of one method, then use Table 3.1 to compare all four at once.
InterviewsClick to learn more
InterviewsClick to learn more
QuestionnairesClick to learn more
SurveysClick to learn more
Table 3.1 sets the four methods side by side on seven features: response rate, cost, interviewer bias, geographic reach, the use of visual aids, the literacy required of respondents and the handling of sensitive questions. The last row deserves particular attention. When people answer a question aloud to another person, many feel some pressure to give the answer that seems acceptable or expected rather than the one that is true. Answers given to an interviewer may therefore be shaded toward what the interviewer seems to expect. For example, a respondent asked face to face how many alcoholic drinks they have in a week may report fewer than they actually drink. Because no one is listening or watching, questionnaires completed in private by mail or online remove much of this pressure, and so they tend to produce more candid answers to sensitive questions.
Table 3.1. Comparison of in-person, telephone, mail and internet administration of questionnaires across seven features.
| Feature | In-Person | Telephone | Internet | |
|---|---|---|---|---|
| Response Rate | High | Moderate–High | Low–Moderate | Low–Moderate |
| Cost | High | Moderate | Low | Low |
| Interviewer Bias | Possible | Reduced | None | None |
| Geographic Reach | Limited | Moderate | Wide | Wide |
| Visual Aids | Yes | No | Printed images only | Yes |
| Literacy Required | No | No | Yes | Yes |
| Sensitive Questions | Answers may be shaded toward what the interviewer seems to expect | Some under-reporting of sensitive behaviours | Self-completed in private; answers tend to be more candid | Self-completed in private; answers tend to be more candid |
No method wins on every row, so choosing one for a particular study means working through the rows of Table 3.1 in turn. Ask who the respondents are and how they can be reached, whether they can read the form, how sensitive the questions are, and what the budget allows. A survey of parents across a region, for example, might use a mailed or web questionnaire to keep costs low and let parents answer questions about illness in private, with telephone follow-up for those who do not respond. Mixed designs of this kind are common because they combine the strengths of more than one method.
This section has defined questionnaires and surveys, set out the steps of planning, and compared the types of questionnaire and the methods of administering them. The key takeaways below summarise these points, and the next section turns to the choice between open and closed question formats.
Key Takeaways
- 1.1 People often use the words questionnaire and survey as if they meant the same thing, but in epidemiology they name two different things. A questionnaire is the data-collection tool: the structured set of questions you put in front of respondents, whether in person, by telephone, by mail or online. A survey is the observational study that uses a questionnaire to collect descriptive information about a population, such as how common a disease or an exposure is. The simplest way to keep them apart is to remember that the questionnaire is the instrument and the survey is the study design. When a question asks you to distinguish them, give each its role in a sentence, and remember that survey-level decisions, such as how the sample was drawn, affect how you can interpret the answers the questionnaire collects.
- 1.2 Good questionnaires are planned long before the first question is written. Start by defining the study objectives: what you actually need to know, and which variables must be measured to answer your research questions. Then consult the people who matter, including subject experts, the end users of the data (such as policymakers) and members of the population you will survey; the Delphi technique is a structured way to build consensus among these interest holders. Focus groups of 6 to 12 people from the study population, guided by an independent moderator, reveal the attitudes, concerns and everyday terms that respondents use. Before drafting new items, search for existing validated instruments, such as the Canadian Community Health Survey, because reusing validated questions saves time and lets you compare your sample with national or provincial norms. If you are asked how to plan a questionnaire, give these steps in order, say what each one adds, and avoid the common trap of starting with item writing.
- 1.3 Questionnaires come in two broad types, and the stage of your research decides which one you need. A qualitative (explorative) questionnaire uses mainly open questions that let people express their views freely; it suits the hypothesis-generation phase, when you are still finding out which issues matter, and it is often administered through interviews that may be recorded. A quantitative (structured) questionnaire uses mostly closed questions to capture specific variables whose answers can be coded and analysed statistically, and it is the main design in epidemiological research. In practice a qualitative phase often comes first and identifies the ideas that the quantitative questionnaire then measures. When you are asked to choose between them, look at the aim of the study: exploring what matters points to a qualitative questionnaire, while measuring how common something is or comparing groups points to a quantitative one.
- 2.1 No method of administration is best on every count, so choosing one is always a matter of weighing trade-offs. In-person interviews usually achieve the highest response, allow visual aids and suit respondents with low literacy, but they are expensive, geographically limited and open to interviewer bias, and answers to sensitive questions may be shaded toward what the interviewer seems to expect. Telephone interviews keep many of these advantages at moderate cost with less interviewer bias, but respondents will give only limited time and some under-report sensitive behaviours. Mailed and internet questionnaires are cheap, reach wide areas and have no interviewer bias, and because people complete them in private their answers to sensitive questions tend to be more candid; however, response is lower, respondents must be able to read, and web surveys reach only people with internet access. When a question describes a study, work through the rows of Table 3.1 (who the respondents are and how they can be reached, whether they can read the form, how sensitive the topic is and what the budget allows) and justify your choice on the features that matter most for that study. Mixed approaches, such as a mailed or web questionnaire with telephone follow-up for non-responders, are common and often make the strongest answer.
1. A health authority plans to estimate how many adults in the region smoke. It will select 1,200 adults at random and ask each of them to complete a 20-item form about tobacco use. A student writes in the project plan: “Our survey is the 20-item form; the questionnaire is the plan to sample 1,200 adults and estimate how common smoking is.” What is wrong with the student’s statement?
2. A city asks your team for a questionnaire on food insecurity among university students. At the first meeting, a team member proposes to spend the afternoon writing 40 new items. According to the lesson, what should come before any new item is written?
3. Little is known about why young adults in a rural community avoid the local sexual-health clinic. Before the team measures how common each barrier is, it wants to find out what the barriers are. Which type of questionnaire suits this first phase?
4. A provincial programme wants to ask 5,000 adults across the province about alcohol use and sexual behaviour. The budget is small, and nearly all of these adults read well and have internet access. Using Table 3.1, which administration method fits best?
5. A regional health authority will estimate how many adults aged 65 and older had a fall in the past 12 months, and it will repeat the estimate every two years. Early focus groups show that many older adults do not count a slip or a stumble that causes no injury as a “fall”. What should the main questionnaire be, and how should it ask about falls?
6. The same health authority has $9,000 for data collection and needs about 300 completed questionnaires. About 4 in 10 older adults in the region do not use the internet, many have poor eyesight or limited reading ability, and most live in small, widely spread communities. A pilot gave the cost per completed questionnaire as: in-person interview $40, telephone interview $22, mailed questionnaire $8, web survey $5. Which choice is best supported?
✦ Pass the knowledge check with 100% to continue
Designing Questions: Open & Closed Formats
⏱ Estimated reading time: 15 minutes
Overview
Section 1 covered the up-front planning that shapes the entire instrument. This section zooms in on the individual question, and in particular on the choice between open-ended and closed-ended formats, arguably the most consequential design decision after objective-setting. It begins with what happens cognitively when a respondent answers any question at all, because that process explains the design rules that follow. It then covers open questions, the four main types of closed question and the trade-off between the two formats, and it ends with the rule that the categories of a multiple-choice question must be mutually exclusive and jointly exhaustive, the rule that draft items break most often.
Learning Objectives
- 3.1 Describe the four cognitive steps involved in responding to a question, and identify the step at which a flawed item fails.
- 4.1 Explain when and how to use open questions, including why exact numbers such as age are preferred to ranges and why income is the usual exception.
- 4.2 Distinguish among the four main types of closed questions (checklist, multiple-choice, rating and ranking) and state the design rules for each.
- 4.3 Identify the advantages and limitations of closed versus open question formats.
- 5.1 Recognise multiple-choice categories that overlap or leave gaps, and repair them so that they are mutually exclusive and jointly exhaustive.
3.1 The Four Cognitive Steps of Answering a Question
When a respondent encounters a question, they go through four distinct cognitive steps. Understanding these steps, formalised in the cognitive model of the survey response process (Schwarz, 1999; Tourangeau & Yan, 2007), helps researchers design questions that produce accurate, reliable answers. Figure 3.1 draws the steps as a sequence in which a question can fail at any point.
The first step is understanding: will the respondent understand the question? By this we mean that the respondent must take from the words the same meaning the researcher intended. If the wording is technical or ambiguous, different respondents will in effect answer different questions, so the question must be clearly worded in non-technical language. The second is retrieval: will the respondent know, or be able to recall, the answer? Even a question that is perfectly understood is useless if the information is not stored in the respondent's memory or is too hard to bring back. If the question requires information they do not have, they might skip it or fabricate a response, and a made-up answer is harder to detect than a blank one. The third is judgement: once the information is recalled, the respondent must turn it into an answer. If the question involves a subjective decision, such as an opinion or belief, each respondent may apply a different standard, so the designer should ask whether there is a way to make it less subjective. For example, asking how many days a person exercised in the past week requires less personal judgement than asking whether they are "active". Special care in question design is needed to capture such information reliably. The fourth is communication: the respondent must fit the answer they have formed into the options the questionnaire provides. This step asks whether the possible responses are clear, with an appropriate method of recording the answer, because a respondent who knows the true answer still cannot report it if no option matches.
Once you have drafted a question, these four steps become four questions to ask of it. Will the respondent understand it? Will they know the answer? If it requires judgement, can you make it more objective? Are the response options clear and appropriate? To diagnose a flawed item, read it as a respondent would and stop at the first step where an honest respondent would struggle, because that step tells you what to fix. Worked Example 3.1 applies this routine to one faulty item for each of the four steps; for each item, watch for the first point at which an honest respondent would get stuck and how the fix targets that point.
Worked Example 3.1: At Which Step Does the Item Fail?
- Understanding. “Have you had an episode of acute gastroenteritis this month?” Many respondents will not know the medical term. Fix: use plain words, such as “Have you had diarrhea or vomiting since 1 March?”
- Retrieval. “How many cups of coffee did you drink on each day of the past year?” Nobody can recall a daily count over a year, so the respondent skips the item or makes up a number. Fix: shorten the reference period, such as “On a typical day in the past 7 days, about how many cups of coffee did you drink?”
- Judgement. “Is your diet healthy?” Each respondent must decide privately what “healthy” means, so answers are not comparable. Fix: ask about a concrete behaviour, such as the number of days in the past 7 on which the respondent ate vegetables.
- Communication. “How often do you exercise? (Yes / No)” The respondent may know the answer, but the options cannot record a frequency. Fix: offer response options that match the question, such as a count of days in the past 7.
In general, a question that asks for too much detail over too long a period fails at retrieval, and the remedy is a shorter reference period or a “typical day” in a recent week.
The example shows that naming the failing step is what points to the fix: plain words for understanding, a shorter reference period for retrieval, a concrete behaviour for judgement and matching options for communication. These four steps apply to every question, whatever its format, and the rest of this section turns to the formats themselves, beginning with the open question.
4.1 Open Questions
Open questions (also called open-ended questions) place no restrictions on the type of response expected. They are more commonly used in qualitative research, because the responses may not be easily standardised for statistical analysis, but they can be valuable in quantitative research as well. Their most important use there is the fill-in-the-blank question for numerical data, and one rule governs it: capture an exact number wherever respondents will give one, because continuous data can always be grouped later but a range can never be turned back into an exact value. Income is the usual exception, because people are more willing to choose a range than to state an exact figure. Expand the four panels below for the details of each use of open questions in quantitative research, including how to word an income item and why the units must always be stated.
One common type of open question in quantitative research is the fill-in-the-blank question, used especially for capturing numerical data. For example: “How many people live in this household? ___”
Where possible, capture numerical data as a continuous variable rather than using pre-defined ranges. Knowing someone is exactly 47 years old is more informative than knowing they fall in the 40–59 age bracket. Continuous data can always be categorised later during analysis. By this we mean that an exact age can be sorted into whatever groups the analysis needs, whether 40–59, 45–54 or simply "over 50", whereas once a respondent has ticked "40–59" there is no way to recover whether they were 41 or 58. Collecting the exact value therefore keeps every later option open.
When seeking sensitive information (such as total family income), respondents may be more willing to indicate a category or range rather than an exact value. In these cases, using ranges is an acceptable compromise to encourage honest responses.
Income is therefore the usual exception to the preference for exact numbers. Asking “What was your household’s income last year? $____” invites refusals and guesses. A better item asks which range best describes the income, offers ranges that do not overlap and together cover every possible income, and ends with “Don’t know” and “Prefer not to answer”. Worked Example 3.5 in Section 4 shows a complete example, and income items belong near the end of the questionnaire with the other sensitive questions.
When capturing numerical data, always specify the units being used (kilograms, pounds, centimetres, etc.) and consider offering a choice of measurement scales to accommodate different respondents. The reason is that a bare number is uninterpretable without its unit. For example, a recorded weight of 70 could mean 70 kilograms or 70 pounds, which describe very different people, and the analyst cannot tell which was meant. Because some respondents know their weight in pounds and others in kilograms, letting them choose the unit, and recording which one they chose, avoids both guessing and conversion errors.
Sometimes a “comments” section can be attached to a closed question to allow respondents to elaborate or express their opinion. This combines the advantages of both question types and can capture unexpected information.
Open questions therefore earn a place even in a structured questionnaire, both for exact quantities and, through a comments section, for answers the designer did not anticipate. Most items in a quantitative questionnaire, however, are closed questions.
4.2 The Four Types of Closed Question
In closed questions (also called closed-ended questions), the respondent selects from a pre-defined range of options. Closed questions are generally easier for respondents to answer and easier to code for data entry. They come in four main types, and each follows its own design rules: a checklist allows several answers, a multiple-choice question allows only one, a rating question records an opinion or judgement on a scale, and a ranking question records an order of priority. Open each tab below to see an example of each type and the rules for designing it.
Checklist Questions
In a checklist question, the respondent checks all options that apply. The options do not need to be mutually exclusive or jointly exhaustive. This is because each option is treated as a separate yes/no variable in the database. By this we mean that a list of five symptoms produces five columns in the dataset, one per symptom, and each respondent is coded "yes" or "no" in every column. A respondent who ticks both headache and fever simply has "yes" in two columns, so an overlap between options causes no problem.
Example: “Which of the following symptoms have you experienced? (Check all that apply)”, followed by a list of symptoms with checkboxes.
This is equivalent to having a series of individual yes/no questions for each category.
Two-Choice and Multiple-Choice Questions
Multiple-choice questions require categories that are mutually exclusive (no overlap) and jointly exhaustive (cover all possibilities). Adding an “Other (please specify)” category ensures exhaustiveness.
It is recommended to limit choices to 5 options for in-person/telephone interviews and 10 for mailed/internet questionnaires. The limit is lower when questions are spoken because a listener must hold every option in memory, whereas a reader can look back over a written list. In addition to the number of options, be aware of order effects. By this we mean that the position of an option in the list can change how often it is chosen, regardless of its content. In written, self-administered lists respondents tend to favour options near the top (a primacy effect), because they read from the top and may stop once they reach a plausible answer. In questions read aloud, by contrast, they tend to favour the last options they hear (a recency effect), because those options are freshest in memory when the interviewer stops speaking. Both, together with acquiescence (the habit of simply agreeing), are forms of satisficing, where a respondent settles for a good-enough answer rather than the most accurate one (Krosnick, 1991). Satisficing is not dishonesty; it reflects the fact that working out the most accurate answer takes effort, and a tired or uninterested respondent takes a shortcut. Varying the option order across different versions of the questionnaire can help mitigate these effects, because any advantage of being first or last is then spread across all the options rather than always favouring the same one.
Example: “How do you draw water from the cistern? (Select one)”, with options like “With a bucket,” “With a manual pump,” “With an electric pump,” “Other (please specify).”
Breaking this rule is the most common fault in multiple-choice items, and 5.1 shows how to test categories and repair them.
Rating Questions
Rating questions ask the respondent to assign a value on a pre-defined scale. The most common form is the Likert scale (Likert, 1932), where respondents indicate their level of agreement (e.g., strongly agree, agree, neutral, disagree, strongly disagree). Scale design choices, including number of points, midpoint, and label wording, affect both reliability and the likelihood that respondents will satisfice (settling for a satisfactory rather than an optimal answer; Krosnick, 1991).
Key considerations:
- Use a minimum of 5 to 7 categories to avoid serious loss of information.
- Decide whether to include a neutral midpoint or use a “forced-choice” scale (even number of categories). A midpoint such as “neutral” gives a place to respondents who genuinely have no view, but it also offers an easy resting place for those who are satisficing. A forced-choice scale removes that resting place by making every respondent lean one way or the other, at the cost of pushing truly neutral people into a side they do not hold.
- Always include a “don’t know” or “not applicable” option to distinguish non-responses from true neutrality. By this we mean that a person who has thought about an issue and feels neither for nor against it is different from a person who has no knowledge of the issue at all. Without a “don’t know” option, both would tend to choose the midpoint, and the two very different answers would be recorded as the same. Krosnick (1991) noted that a respondent who is satisficing may also say “don’t know” to avoid the effort of forming an opinion, so a large share of “don’t know” answers is worth examining in its own right.
- Parametric statistics (mean, standard deviation) should only be computed if there are at least 5 points on the scale and the points can be assumed to be equally spaced. A mean treats the step from “disagree” to “neutral” as the same size as the step from “neutral” to “agree”. If respondents do not experience the steps as equal, an average of the coded values can describe the group misleadingly.
A Visual Analogue Scale (VAS) is a special type of rating question where respondents mark a point on a continuous line between two extremes (e.g., “terrible” to “excellent”). VAS is widely used for pain measurement in clinical research.
Ranking Questions
Ranking questions ask respondents to order options by priority or importance. They can be difficult to complete, especially when the list is long, because all categories must be held in mind simultaneously.
Important limitations:
- Rank intervals are unknown: the difference between ranks 1 and 2 may not equal the difference between ranks 2 and 3. For example, a respondent may care a great deal about their first and second priorities and hardly at all about their third, yet the ranks record only the order, not how far apart the items are in importance.
- Respondents may assign “tied” ranks, and decisions about how to handle these must be made in advance.
- If categories are omitted, it may influence how respondents rank the remaining options.
- For in-person interviews, using physical cards that respondents can sort may simplify the ranking process.
Choosing among the types starts with two questions: should the respondent give one answer or several, and does the study need an amount, an opinion or an order of priority? Once the type is chosen, its rules follow, such as at least 5 points and a “don’t know” option for a rating scale. Whatever the type, a closed question trades some richness for efficiency, which raises the question of when an open format is the better choice.
4.3 Weighing Open Against Closed Formats
The choice between open and closed formats is a trade-off between richness and efficiency. Open questions capture richer, more detailed data in the respondent’s own words, including answers the designer did not anticipate, but they are harder to analyse because the answers must be coded afterwards. Closed questions are efficient to code and analyse, but they risk oversimplifying issues or forcing respondents into categories that do not fully represent their views (Krosnick, 1999). The best questionnaires often use a thoughtful combination of both: closed items for the variables being formally measured, and open elements, such as an “Other (please specify)” line or a comments section attached to a closed question, to catch what the closed options miss. When comparing the formats for a particular study, link the choice to its purpose, since exploration favours open questions while measuring and comparing favour closed ones.
When a closed format is chosen, it works only if its categories fit every respondent, and the rule for multiple-choice categories is where draft items most often fail.
5.1 Mutually Exclusive and Jointly Exhaustive Categories
For a multiple-choice question to work, every respondent must find exactly one category that fits. Two properties guarantee this, and they are worth committing to memory.
The Rule for Multiple-Choice Categories
The categories must be mutually exclusive (no overlap, so that no value fits two categories) and jointly exhaustive (covering all possibilities, so that every value fits one). Adding an “Other (please specify)” category helps to ensure exhaustiveness.
A checklist is exempt from the rule, because each of its options is a separate yes/no variable, but every multiple-choice item must pass it. The quickest check is to test the boundary values and the extremes of the range. Worked Example 3.2 applies that check to a common faulty age item; watch for the values that fit two categories, the values that fit none, and the final check that the repair has not created a new fault.
Worked Example 3.2: Repairing Overlapping and Incomplete Categories
A draft questionnaire asks: “What is your age? ☐ 18 to 30 ☐ 30 to 45 ☐ 45 to 65”.
Step 1, test the boundaries. A 30-year-old or a 45-year-old fits two categories, so the categories are not mutually exclusive.
Step 2, test the extremes. A 17-year-old or a 70-year-old fits none, so the categories are not jointly exhaustive.
Step 3, repair the item. The better item asks “How old are you, in years? ____”. An exact age can be grouped later in any way the analysis needs, which is the preference for exact numbers from 4.1. If categories must be used, make them non-overlapping and complete for the population surveyed, such as, for a survey of adults, 18 to 29, 30 to 44, 45 to 64, and 65 or older.
Step 4, check the repair. Test the new boundaries again: 29 and 30, 44 and 45, and 64 and 65 each fall in exactly one category, and the open top category leaves no gap above 64.
The example shows that one item can break both properties at once, so a repair that removes only the overlap, or only the gap, fixes half the problem. Testing boundaries and extremes, before and after the rewrite, is the routine to use on any multiple-choice item.
This section has described the four cognitive steps behind every answer and the design rules for open and closed questions. The key takeaways below summarise these points, and the next section turns to the wording, structure and pre-testing of questions.
Key Takeaways
- 3.1 Every time someone answers a question, they move through four cognitive steps: understanding what is asked, retrieving the information needed, making a judgement when the question asks for an opinion or estimate, and communicating the answer through the response options you offer. A well-designed question supports all four steps, and a flawed one breaks down at one of them, most often retrieval or judgement. To diagnose a draft item, read it as a respondent would and stop at the first step where an honest person would struggle: a medical term such as “acute gastroenteritis” fails at understanding, a daily count of coffee over a whole year fails at retrieval, “Is your diet healthy?” fails at judgement, and Yes/No options for “How often do you exercise?” fail at communication. Naming the step tells you what to fix; for a retrieval failure, for example, the remedy is a shorter reference period or a “typical day” in a recent week. In an exam answer, name the step, explain briefly why the respondent would struggle there, and offer a rewrite that removes the problem.
- 4.1 Open questions place no limits on how someone can answer, which makes them the natural choice for exploratory work and for capturing answers you did not anticipate, for example through a comments section attached to a closed question. In quantitative questionnaires their most useful form is the fill-in-the-blank, such as “How old are you, in years? ____”, which records an exact number rather than a pre-set range. Exact numbers are preferred because you can always group continuous data into categories later, but you can never recover the precision lost by asking for a range: knowing someone is 47 tells you more than knowing they are in the 40 to 59 bracket. Income is the usual exception, because people are more willing to choose a range than to state an exact figure, so a good income item offers non-overlapping ranges that cover every possible income, followed by “Don’t know” and “Prefer not to answer”. When you write a numerical item, also state the units (kilograms or pounds, for example), and when you see a sensitive quantity such as income asked as an exact amount, flag it as the exception and rewrite it in ranges.
- 4.2 Closed questions give respondents a fixed set of options, and each of the four main types follows its own rules. In a checklist (“check all that apply”) the options may overlap, and each option becomes a separate yes/no variable in the dataset. A multiple-choice question allows only one answer, so its categories must be mutually exclusive and jointly exhaustive, usually with an “Other (please specify)” option, and it works best with no more than about 5 options in an interview and 10 on a mailed or online form. A rating question, most often a Likert scale, needs at least 5 points and a “don’t know” or “not applicable” option so that genuine uncertainty or non-response is not confused with a neutral answer, and means and standard deviations make sense only when the points can be assumed to be equally spaced. A ranking question asks people to put options in order, which is demanding for long lists and produces ranks whose intervals are unknown, so the gap between first and second may not equal the gap between second and third. When you choose a format, ask whether respondents should give one answer or several and whether you need an amount, an opinion or an order of priority, and then apply the rule for that type.
- 4.3 Choosing between open and closed formats is a trade-off between richness and efficiency. Open questions capture detailed and unexpected answers in the respondent’s own words, but those answers have to be coded afterwards and are harder to analyse statistically. Closed questions are quicker to answer and easy to code and analyse, but they can oversimplify an issue or force people into categories that do not fit their situation. Well-designed questionnaires usually combine the two, using closed items for the variables being formally measured and open items, such as an “Other (please specify)” line or a comments box, to catch what the designer did not anticipate. When asked to compare the formats, give one advantage and one limitation of each and link your choice to the purpose of the study: exploration favours open questions, while measuring and comparing favour closed ones.
- 5.1 For a multiple-choice question to work, every respondent must find exactly one category that fits. That means the categories must be mutually exclusive, so that no value fits two categories, and jointly exhaustive, so that every possible value fits one; an “Other (please specify)” option helps with the second rule. The quickest check is to test the boundary values and the extremes: in “18 to 30 / 30 to 45 / 45 to 65”, a 30-year-old fits two categories, while a 17-year-old or a 70-year-old fits none. To repair the item, ask for the exact number where you can (“How old are you, in years? ____”), or, if categories are needed, make them non-overlapping and complete for the population surveyed, such as 18 to 29, 30 to 44, 45 to 64 and 65 or older. In an exam answer, name both faults when both are present, give a value that shows each one, and check that your rewrite does not create a new gap or overlap.
1. A draft item for a study of physical activity reads: “On each day of the past 6 months, how many minutes did you spend walking?” At which cognitive step is this item most likely to fail?
2. A draft household questionnaire asks each adult’s age with the categories ☐ Under 20 ☐ 20 to 39 ☐ 40 to 59 ☐ 60 or older, and asks for household income as an exact dollar amount, $______. Which advice for these two items follows the lesson?
3. A clinic questionnaire asks: “Which of the following symptoms have you had in the past 7 days? Tick all that apply: ☐ Cough ☐ Fever ☐ Sore throat ☐ Runny nose.” What type of closed question is this, and how should it be entered in the dataset?
4. A draft item reads: “How many times did you visit a doctor in the past 12 months? ☐ 1 to 3 ☐ 3 to 5 ☐ 6 to 10”. Which statement correctly describes the fault in the response categories and a repair?
5. A team evaluating a new walk-in clinic wants to ask patients, “What is the main reason you chose this clinic today?” The team needs to count how often each reason is given, but it is not sure it can foresee every reason. Which question format best balances the strengths and weaknesses of open and closed questions?
✦ Pass the knowledge check with 100% to continue
Wording, Structure & Pre-Testing
⏱ Estimated reading time: 14 minutes
Overview
Sections 1 and 2 settled the larger questions of planning and question format. This section is about the small choices that determine whether respondents understand a question the same way you do. Wording, structure, layout and pre-testing are where most of the residual measurement error in surveys is introduced, or eliminated. The section sets out four rules for wording a question and a routine for using them to repair a draft item, then the structure and layout of the questionnaire as a whole, and finally the pre-testing and validation that check whether the finished instrument works and measures what it should.
Learning Objectives
- 5.2 Apply the four wording rules (short and simple, specific, not double-barrelled, not leading) to name the rule a draft item breaks and rewrite it.
- 6.1 Describe how to structure and lay out a questionnaire: an introduction, easy opening items, the funnel approach, sensitive items near the end and pre-coded responses.
- 6.2 Explain the importance of pre-testing and validation, and state what expert review, field pre-testing, think-aloud interviews, test-retest reliability and comparison with a gold standard can each reveal.
5.2 Four Rules for Wording Questions
The way a question is worded has a major impact on the validity of the results. Poorly worded questions can confuse respondents, introduce bias, or yield uninterpretable data (Schwarz, 1999). Four rules guard against these problems: keep each question short and simple, make it specific, avoid double-barrelled questions that ask about two things at once, and avoid leading questions that suggest a desired answer. Expand each rule below for its details and an example of an item that breaks it.
Questions should rarely exceed 20 words. Avoid abbreviations, jargon, and complex technical terminology. Always consider the respondent’s level of technical knowledge.
Poor: “How many cases of acute gastrointestinal illness occurred during the time period?”
Better: “How many times did you have diarrhea during January?”
Make questions as specific as possible. If asking about a health event, specify the time frame (e.g., “over the past month”) and clearly define the event. For example, define what constitutes a “new case” of diarrhea: perhaps at least 2 days of normal bowel movements before onset counts as a new episode.
Choose a reference period that respondents can actually remember. A recent, short window (the past 7 days, or a typical day in the past week) gives more accurate answers than a daily count over a month or a year, which fails at the retrieval step described in Section 2.
A double-barrelled question asks about two things at once. For example: “Do you think traveller’s diarrhea is a serious problem that people should get vaccinated for?” is really asking two questions: one about the seriousness of the disease and one about vaccination. The problem is that a single answer cannot tell you which part the respondent was responding to. For example, a person who thinks the disease is serious but does not want vaccination has no honest way to answer, and if they answer “no”, the researcher cannot tell whether they rejected the seriousness, the vaccination, or both. These should therefore be separated into two distinct questions, each of which can be answered on its own.
Leading questions suggest a desired answer and can bias responses. By this we mean that the wording signals which answer the researcher expects or approves of, and many respondents, wishing to be agreeable, move toward that answer. Because the push is always in the same direction, the error does not cancel out across respondents; it shifts the whole result. For example: “Should people have to suffer from regular bouts of diarrhea because of a bad water supply in this region?” implies that the answer should be “no.”
A more neutral alternative: “Do you think the water supply in this region needs to change?”
Two of the rules call for a short procedure rather than a single change. Being specific means naming the time frame and also defining what counts as one occurrence of the event, so that every respondent counts episodes the same way. Without that definition, one long illness can be counted as a single episode by one respondent and as several by another. The definition goes into the item itself as a counting instruction, for example “Count a new time only after at least 2 days of normal bowel movements.” Neutralising a leading item also takes a few steps. Remove the loaded premise or emotive wording (such as “suffer” or “Given the well-known dangers of ...”), and drop openings that presuppose the answer, such as “Do you agree that” or “Don’t you think”. Then either name both directions (“Do you support or oppose ...?”) or ask a plain open question such as “Do you think the water supply in this region needs to change?”, and offer balanced response options, for example from strongly support to strongly oppose. As with any repair, check that the rewrite does not add a new fault.
The rules are most useful as a routine for reviewing draft items, in three steps. First, read the item as a respondent and note the cognitive step at which you would struggle. Second, name the rule the item breaks, in a short phrase. Third, write an improved version and check that it removes the fault you named, without adding a new one. Worked Example 3.3 applies this routine to six draft items; watch for items that break more than one rule, for the episode definition written into the diarrhea item, and for the way the leading item is neutralised, because the rewrite must then remove every fault.
Worked Example 3.3: Diagnosing and Repairing a Draft Item
- “Do you agree that the clinic’s staff are friendly and efficient?” Rules broken: double-barrelled (friendly and efficient are two judgements) and leading (“Do you agree that” invites agreement). Improved: two items, “How friendly were the clinic staff?” and “How efficient was the clinic’s service?”, each with a labelled five-point scale.
- “Do you often feel tired?” Rule broken: not specific (“often” is vague and there is no time frame). Improved: “In the past 14 days, on how many days did you feel too tired to do your usual activities? ____ days”.
- “How many episodes of nocturnal enuresis did your child experience?” Rules broken: not short and simple (medical jargon) and not specific (no time frame). Improved: “In the past 30 days, on how many nights did your child wet the bed? ____ nights”.
- “How many cases of acute gastrointestinal illness occurred during the time period?” Rules broken: not short and simple (“acute gastrointestinal illness” and “cases” are jargon) and not specific (no time frame and no definition of an episode). Improved: “In the past 30 days, how many times did you have diarrhea? Count a new time only after at least 2 days of normal bowel movements. ____ times”.
- “Should people have to suffer from regular bouts of diarrhea because of a bad water supply in this region?” Rule broken: leading (“have to suffer” and the premise about a bad water supply imply that the answer is “no”). Improved: “Do you think the water supply in this region needs to change?”, with the emotive premise removed and nothing in the wording that points to one answer.
- “How many hours of television do you watch? ☐ 0 to 2 ☐ 2 to 4 ☐ 4 to 6” Rule broken: the categories are not mutually exclusive (2 and 4 hours fit two categories) or jointly exhaustive (more than 6 hours fits none), and there is no time frame. Improved: “On a typical day in the past 7 days, about how many hours of television did you watch? ____ hours”.
On an examination, an answer earns credit when it names the specific rule (not just “unclear”) and gives a rewrite that actually removes that fault.
The example shows that faults often come in pairs, and the last item also breaks the category rule from Section 2 (5.1), a reminder that wording and format problems occur together. Once each item is well worded, the items must be arranged in a logical order and presented on a usable form.
6.1 Structure and Layout of a Questionnaire
A well-structured questionnaire guides the respondent through a logical flow and creates a positive experience that encourages completion. Three elements frame that flow. The questionnaire should begin with an introduction that explains the rationale and importance of the questionnaire, how the data will be used and the assurance of confidentiality, and that estimates how long it will take to complete. The opening questions should build confidence: easy, non-threatening items that put the respondent at ease. The instructions should be clear and concise, with key instructions highlighted in bold or other formatting, remembering that people read instructions only when they think they need help.
Within the body of the questionnaire, questions should be grouped in sections, either by subject (illness, water supply, housing) or chronologically (current health, past week, past month, previous illness). Within each section, questions should follow the funnel approach, starting with broader, more general questions and gradually becoming more specific and focused, which helps respondents warm up to the topic before being asked for detailed information. For example, a section on water might first ask where the household gets its drinking water, then how the water is stored, and only then how often it is boiled before drinking. In addition, pairs of questions that capture essentially the same information can be included at different points to verify responses and check internal consistency. By this we mean that the same fact is asked twice in different ways, for instance age early in the form and year of birth later on. If the two answers agree, the researcher has more confidence in them; if they conflict, the response is flagged as possibly careless or misunderstood. Sensitive items, such as income, sexual behaviour or substance use, belong toward the end of the questionnaire. By then the respondent has built rapport and seen how the information is used, and a respondent who stops at a sensitive item has already answered everything before it. A questionnaire that opens with its most sensitive item reverses this logic and invites people to stop at the very start.
For mailed or online questionnaires, the layout of the form matters as well, because visual appeal and ease of completion are essential. Three considerations stand out: pre-coding the responses, a professional appearance and an optimal length. Click each card below to see what each involves and how it helps.
ResponsesClick to learn more
AppearanceClick to learn more
LengthClick to learn more
Each of these makes the form easier to complete or to enter, and length returns in Section 4, because shorter questionnaires consistently achieve higher response rates. However carefully a questionnaire is worded, ordered and laid out, it can still contain problems that only testing reveals.
6.2 Pre-Testing and Validation
Every questionnaire must be checked before it is used, and the checking answers two different questions. Pre-testing asks whether the questionnaire works for real respondents, and validation asks whether it measures what it is intended to measure.
6.2.1 Pre-Testing Methods
All questionnaires must be pre-tested before deployment. Pre-testing identifies confusing, ambiguous, or misleading questions, layout problems, and issues with instructions, and it also serves to estimate completion time (Presser et al., 2004). Four methods are used, and each catches a different kind of problem. Open each tab to see what a method involves and what it reveals.
Expert Review
The first step is to have colleagues or subject-matter experts review the questionnaire. They can evaluate whether all important issues are covered and identify obvious problems with clarity or logic.
Field Pre-Test
Pre-testing on a small sample of the target population is essential. Have respondents complete the questionnaire as they would in the actual study. This reveals questions that are hard to understand, unanswerable, or that need additional response categories.
Think-Aloud Pre-Test
In a think-aloud pre-test, the respondent narrates their thought process as they work through each question, the core technique of cognitive interviewing (Beatty & Willis, 2007). This provides direct insight into how questions are being interpreted and where confusion or difficulty arises.
Test–Retest Reliability
If feasible, re-administer the questionnaire to the same group of respondents after an appropriate time interval. The interval should be long enough that they do not recall their original answers, but short enough that the underlying information has not changed. Both limits matter for the same reason: the comparison is meant to show whether the question itself produces the same answer twice. If the interval is too short, respondents may simply repeat what they remember writing, which makes the question look more repeatable than it is. If it is too long, real changes in the respondent's health or behaviour will make the answers differ even when the question is working well. This approach assesses the repeatability of the questions. Note: it is only valid if the questionnaire itself has not been modified between administrations.
Matching the method to the problem is the practical skill. Expert review checks coverage and logic from the designer’s side, a field pre-test shows which items real respondents cannot answer and how long the form takes, a think-aloud interview reveals how respondents actually interpret an item, and a test-retest comparison shows whether answers repeat. Repeatable answers are not necessarily correct ones, however, and that is the question validation addresses.
6.2.2 Validation and the Cost of Unreliable Measurement
Validation assesses whether the questionnaire actually measures what it is intended to measure, and four approaches can be used. The most direct is comparison with a gold standard, in which questionnaire responses are compared with directly measured quantities. A gold standard is the best available measure of the true value, one we are prepared to treat as correct. For example, dietary intake reported on a food frequency questionnaire can be compared against actual measured intake, such as food weighed and recorded over several days. If the two agree closely, the questionnaire can be trusted to stand in for the more costly direct measurement. Because a gold standard is often expensive or impossible to obtain, responses can also be compared with an established method, the results of a well-validated existing instrument. When no external standard exists, repeated administration of the questionnaire to the same respondents assesses its repeatability. Finally, method comparison evaluates whether different administration methods (e.g., mail vs. telephone) produce comparable results.
Repeatable Is Not the Same as Valid
Close agreement between two administrations shows that answers are repeatable, not that they are correct: respondents who always under-report a behaviour will under-report it both times, giving close agreement around the wrong value. Showing validity needs a comparison with a gold standard or an established instrument, and a test-retest comparison is valid only if the questionnaire is unchanged between administrations.
Reliability matters well beyond the questionnaire itself, because unreliable measurement distorts every analysis that uses it. Interactive 3.1 shows why. Picture any measurement as a true signal plus random noise. By this we mean that each recorded answer is made up of the respondent's real value plus a random error that pushes it a little up or a little down, for example because of a misread word or a hurried guess. Reliability is the share of the measured variation that comes from the true signal rather than the noise, so it runs from 0 (all noise) to 1 (no noise). As measurement error grows, reliability falls, and the associations you later estimate with that noisy variable are dragged toward zero. This happens because the random errors are unrelated to the outcome. They spread the measured values out without carrying any of the true relationship, so the real pattern linking exposure and outcome is diluted and looks weaker than it is. For example, if people who truly eat more salt have higher blood pressure, but reported salt intake is full of random error, some heavy salt users will be recorded as light users and the reverse, and the estimated link between salt and blood pressure will shrink. Epidemiologists call that systematic under-statement attenuation bias. One remedy is to add more items to a scale. Because the random errors on separate items tend to point in different directions, averaging several items cancels out some of the noise and pushes reliability back up. Move the sliders below and watch these effects happen together.
📊 Interactive 3.1: Reliability & Measurement Error
Generate 100 respondents with a "true" attitude score; measurements are corrupted by random noise. Watch test-retest correlation, Cronbach’s α (when 5 items are simulated), Cohen’s κ (when scores are dichotomized), and the attenuation of any downstream regression coefficient.
Measurement-1 vs Measurement-2
Effect of error on a regression slope
Outcome Y depends on the true X. The regression on the measured X is attenuated by error.
Interactive 3.1 makes the practical case for pre-testing and validation: a questionnaire with poor reliability does not merely produce noisier answers but also biases the associations estimated from them toward zero.
This section has set out the rules for wording questions, the structure and layout of a questionnaire, and the pre-testing and validation that precede its use. The key takeaways below summarise these points, and the next section turns to response rates and data coding.
Key Takeaways
- 5.2 Small changes in wording can change the answers you get, so four rules guide every item you write. Keep it short and simple: usually under 20 words, in plain language, with no jargon or abbreviations. Be specific: name the time frame, define the event (for example, what counts as a new episode of diarrhea) and choose a reference period that people can actually remember. Ask about only one thing at a time, because a double-barrelled item such as “Do you agree that the clinic’s staff are friendly and efficient?” cannot be answered by someone who feels differently about each part, and avoid leading wording that hints at the expected answer, such as “Should people have to suffer from regular bouts of diarrhea because of a bad water supply in this region?” When you repair a draft item, work in three steps: name the specific rule it breaks (not just “unclear”), write an improved version, and check that the rewrite removes that fault without creating a new one, as the neutral alternative “Do you think the water supply in this region needs to change?” does for the leading example.
- 6.1 The order and look of a questionnaire affect whether people finish it and how accurately they answer. Open with an introduction that explains the purpose, how the data will be used and how confidentiality is protected, and that estimates how long the questionnaire will take; then begin with easy, non-threatening questions that build the respondent’s confidence. Group related questions by subject or chronologically, and within each section use the funnel approach, moving from broad, general questions to more specific ones. Place sensitive items, such as income, sexual behaviour or substance use, near the end, when rapport has been built and a respondent who stops has already answered everything before it. For mailed and online forms, pre-code the responses by printing numerical codes beside the options, keep the layout professional, the instructions clear and the questionnaire as short as possible (ideally under about 1,000 words). When you critique a questionnaire’s structure, check whether it has an introduction, whether it opens with easy items, whether each section moves from general to specific, and whether sensitive questions come too early.
- 6.2 Every questionnaire must be pre-tested before it is used, because problems that are invisible to the designer often become obvious once real people try to answer. Each pre-testing method catches something different: expert review by colleagues checks coverage, clarity and logic; a field pre-test on a small sample of the target population shows which items are hard to understand, go unanswered or need more response categories, and how long the form takes; a think-aloud interview, in which respondents narrate their reasoning as they answer, reveals how each item is actually interpreted; and test-retest administration to the same people on two occasions assesses repeatability, provided the questionnaire is not changed in between. Validation asks a different question: does the questionnaire measure what it claims to measure? The most direct approach compares answers with a gold standard, such as dietary intake from a food frequency questionnaire against directly measured intake, and where no external standard exists you can compare with an established instrument, repeat the administration or compare administration methods. On an exam, match the method to the problem described (for example, a think-aloud interview for an item that respondents seem to read in different ways), and keep the distinction clear: pre-testing checks that the questionnaire works, validation checks that it measures the right thing, and poor reliability drags the associations you later estimate toward zero.
1. A draft item reads: “In the past 7 days, did you wash your hands and use hand sanitiser after riding the bus?” (Yes / No). Which rule of question design does it break?
2. A draft item reads: “Don’t you agree that the city should build more bike lanes to protect cyclists from injury?” Which rewrite removes the fault without adding a new one?
3. A draft questionnaire for a sexual-health study has no introduction. It opens with “How many sexual partners have you had in the past 12 months?”, then asks about the use of sexual-health services, and ends with general questions about health. Which revised order follows the lesson?
4. In a pre-test of a diet questionnaire, the team suspects that respondents read “servings of vegetables” differently: some may count a large salad as one serving and others as three. Which pre-testing method is designed to show how respondents interpret the phrase?
5. A draft item in a web questionnaire for undergraduates reads: “How often do you drink energy drinks and coffee?” Which rewrite removes every fault in the item?
6. To pre-test a questionnaire on sleep, a team gives it to 20 students twice, two weeks apart, and rewords one item between the two rounds. A colleague says: “If the answers agree closely, we will have validated the questionnaire.” Which response is correct?
✦ Pass the knowledge check with 100% to continue
Response Rates & Data Coding
⏱ Estimated reading time: 12 minutes
Overview
Earlier sections covered the design of the questionnaire itself. This section turns to two operational concerns that can make or break a survey: getting people to respond at all, and turning their responses into a clean, analysable dataset. Both deserve as much attention as any decision earlier in the design process. The section explains why low response threatens validity and how to raise it, sets out the principles for coding responses and for deriving analysis variables from them, and ends with the practicalities of data entry and management.
Learning Objectives
- 7.1 Explain why low response threatens validity, and identify design and administrative strategies for maximising response, including follow-up of non-responders.
- 7.2 Explain best practices for coding questionnaire data, including a distinct code for missing answers, separate codes for “don’t know” and refusals, and consistent conventions.
- 7.3 Describe considerations for computer data entry and management, including why coding and data entry are kept separate and why sorting one spreadsheet column can destroy a dataset.
7.1 Maximising Response
7.1.1 Why Low Response Threatens Validity
Regardless of the type of questionnaire used, researchers must make efforts to maximise the response rate. Low response rates increase the risk of selection bias: if non-responders differ systematically from responders, the study results may not accurately represent the target population (Edwards et al., 2009). For example, if people who smoke are less likely to return a health questionnaire than people who do not, the responders will contain proportionally fewer smokers than the population, and the survey will understate smoking prevalence. It is tempting to think that this can be fixed by simply collecting more questionnaires, but that is not the case. Collecting more replies from the same unrepresentative group would make an estimate more precise but not less biased. By this we mean that a larger number of replies narrows the confidence interval around the estimate, but the estimate stays centred on the wrong value, because every extra reply comes from the same kind of person who was already over-represented. This is why the share of the sample that responds matters, not only the number of completed questionnaires.
The term “response rate” refers to the proportion of study subjects who complete the questionnaire. Despite the name, it is a proportion rather than a true rate: a rate carries person-time in its denominator, whereas a response rate is one count divided by another. The label “response rate” is nonetheless universal and well understood in the research community. Read in reverse, the same proportion helps with planning: the number of people to invite is the number of completed questionnaires needed divided by the expected response rate, so 600 completed questionnaires at an expected response of 30% need 600 ÷ 0.30 = 2,000 invitations.
7.1.2 Strategies That Raise Response
Many strategies have been shown to improve response rates, many of them tested across hundreds of randomised trials in the Cochrane methodology review by Edwards et al. (2009). They fall into two groups: design strategies, which concern the questionnaire itself, and administrative strategies, which concern contact with respondents, incentives, and the delivery and return of the questionnaire. Of all of them, follow-up of non-responders with reminders and repeat deliveries is one of the most effective. Open each tab to see the strategies in each group.
Questionnaire Design Strategies
The first group of strategies concerns the questionnaire itself: its stated purpose, layout, pre-testing, length and content.
- Make objectives clear: Explain the purpose of the study clearly to participants so they understand why their participation matters.
- Professional layout: Ensure the questionnaire has a clear structure and professional appearance that signals credibility.
- Pre-test thoroughly: A well-tested questionnaire is easier and less frustrating to complete. Provide an estimate of the time required.
- Minimise length: Keep the questionnaire as short as possible; aim for under 1,000 words. Shorter questionnaires consistently achieve higher response rates.
- Avoid sensitive questions: Unless absolutely necessary, do not ask for sensitive information that might deter participation or trigger social-desirability bias in those who do answer (Tourangeau & Yan, 2007).
Administrative Strategies
The second group concerns contact with respondents, incentives, and the delivery and return of the questionnaire.
- Follow-up contact: Send reminders and repeat deliveries to non-responders. This is one of the most effective strategies.
- Incentives: Offer small incentives for completion. Research shows that financial incentives help, though larger payments do not necessarily produce proportionally higher response rates.
- Include a pen: Including a pen with a mailed questionnaire has been shown to significantly increase response rates (Sharp et al., 2006).
- Return envelope: Provide a stamped, first-class return envelope (not business-reply).
- Advance notice: Notify participants in advance that a questionnaire is coming.
- University sponsorship: Mentioning university or institutional affiliation can increase perceived legitimacy.
- Delivery method: Hand delivery or courier can improve response rates over standard mail.
- Personalise: Address the questionnaire and cover letter personally to the respondent.
- Ink colour: Coloured ink may have a small positive effect on response rates compared with standard black ink.
Several of the design strategies are the design rules of Section 3 seen from the respondent’s side. Worked Example 3.4 applies both groups of strategies to a mailed survey with a low response rate; watch which changes belong to design and which to administration, and which single step matters most.
Worked Example 3.4: The Low-Response Study
A research team mails a 15-page questionnaire to 2,000 households. The questionnaire uses small print, complex medical terminology, and includes a business-reply envelope. After 6 weeks, only 22% of questionnaires have been returned. What design and administrative changes would improve the response rate?
Step 1, design changes. Shorten the questionnaire well below 15 pages, use larger print and a professional layout, replace medical terms with plain language, and pre-test the revised form.
Step 2, administrative changes. One of the most effective steps is to follow up the non-responders with a reminder and a fresh copy of the questionnaire. A stamped, first-class return envelope, advance notice, a personalised cover letter and a small incentive also help.
Step 3, why it matters. With only 22% returned, the responders may differ systematically from the non-responders, so the estimates are at risk of selection bias.
The example shows that a strong answer to a low-response scenario combines both groups of strategies, includes follow-up of non-responders, and links the low response back to selection bias. A good response rate yields completed questionnaires, but they become usable data only once every answer has been turned into a number.
7.2 Coding Questionnaire Data
Before administering any questionnaire, researchers must plan how responses will be coded and entered into a database. Good coding practices ensure data integrity and make analysis more efficient. Some principles apply to every item, and some decisions concern how a single item becomes one or more analysis variables.
7.2.1 Coding Principles
Two principles govern the coding of every item: a unique code for missing values, with separate codes for “don’t know” and refusals, and consistent conventions throughout the questionnaire. Expand each panel below for the details and the mistakes each principle prevents.
Designate a single, unique value to represent missing data (e.g., -999). This value should not be a legitimate answer to any question. Never leave missing values blank, because doing so makes it impossible to distinguish between items the respondent skipped and items the coder missed.
Give “don’t know” and “prefer not to answer” (a refusal) codes of their own as well, distinct from the missing code and from every real answer. A refusal, a genuine “don’t know” and a skipped item mean different things, and the panel “Non-Substantive Answers Need Codes of Their Own” in 7.2.2 explains why. Never code a missing answer as 0, because 0 can be a real answer and would be averaged in with the real values.
Be consistent throughout the questionnaire. For dichotomous (yes/no) variables, always use the same coding convention (e.g., 0 = no, 1 = yes). This avoids confusion during data entry and analysis.
Codes for Answers That Are Not Answers
A skipped item, a “don’t know” and a refusal each get their own code, such as -999, 7777 and 8888, chosen outside the range of real answers. Never use a blank and never use 0, and declare the reserved codes missing in the statistical software before any analysis is run.
A third principle, coding on the form as a step separate from data entry, belongs with data entry in 7.3. Before that, the coding of a single item deserves a closer look, because one answer can become several different variables.
7.2.2 From Response Options to Analysis Variables
The form a question takes on the page and the form it takes in an analysis are two separate decisions. A single item will usually support several different variables, and the choice among them belongs to the analysis plan rather than to the questionnaire. Working that choice out in advance is part of designing the study, because it determines what has to be asked in the first place. An analyst can always collapse detail that was collected. No analyst can recover detail that was never collected.
Income illustrates the point. Section 2 (4.1) established that exact quantities are preferable to pre-set ranges wherever respondents will supply them, and that income is the standard exception, because refusal rates fall when income is asked in bands. Worked Example 3.5 follows one banded income item from the questionnaire into the dataset; watch how the same answer can become five different variables, each with its own use and its own cost.
Worked Example 3.5: One Income Item, Several Analysis Variables
The question as asked. “What was your total personal income, before taxes, in the past 12 months?” (Select one)
- Less than $10,000
- $10,000 to $19,999
- $20,000 to $29,999
- $30,000 to $39,999
- $40,000 to $49,999
- $50,000 to $59,999
- $60,000 to $69,999
- $70,000 to $79,999
- $80,000 to $89,999
- $90,000 to $99,999
- $100,000 or more
- Don’t know
- Prefer not to answer
The variables it can produce. The respondent sees one question with eleven substantive options and two non-substantive ones. The dataset can hold that answer in any of several forms, and the study’s hypotheses determine which form is used. Table 3.2 sets out the most common possibilities for this single item.
Table 3.2. Five ways of coding the banded income item, with the variable each produces, its analytic use and its cost.
| Coding decision | Variable produced | Analytic use | What it costs |
|---|---|---|---|
| Retain the bands as asked | Ordinal variable with 11 substantive levels | Describing the distribution; testing for a trend across bands | Sparse cells at the extremes, and ten coefficients in a regression model, each read against a single reference band |
| Collapse into lower and higher income | Binary variable (0 = higher, 1 = lower) | A single contrast that is easy to communicate and has adequate numbers in each group | Everyone within a group is treated as equivalent, and the cut point carries the entire result |
| Classify against a poverty threshold | Binary variable defined by an external standard | Questions about material deprivation, and comparison with studies using the same threshold | The threshold varies with household size and region, so those items must also appear on the questionnaire |
| Assign the midpoint of each band | Continuous variable measured in dollars | Regression with one coefficient per $10,000 of income, and other parametric methods | Assumes respondents are evenly distributed within each band, and the open top band has no midpoint |
| Reserve values for non-substantive answers | 8888 = prefer not to answer; 7777 = don’t know | Keeps a refusal separable from a genuinely low income and from a coding omission | Must be declared missing in the statistical software before any analysis is run |
None of these codings is correct in the abstract. Each answers a different question, and a well-planned study will often carry more than one of them in the same dataset, with the primary analysis pre-specified and the remainder reported as sensitivity analyses.
The example shows that the coding of an item is an analytic decision with consequences, not a clerical step. Six considerations should inform the choice among the codings in Table 3.2, from the one-way nature of information loss to the companion items that some analyses require. Expand each panel below to read one consideration.
Eleven bands can be collapsed into two categories at any point in the analysis, and the collapse can be revisited as often as the analyst wishes. Two categories can never be expanded back into eleven. The questionnaire should therefore record the finest distinctions respondents will tolerate, and the coarsening should be left to the coding stage, where it remains reversible. This is the principle behind the earlier recommendation to capture exact ages rather than age brackets, applied one step further along the pipeline.
An eleven-level categorical predictor contributes ten coefficients to a regression model, each estimated against a single reference band and each carrying its own confidence interval. By this we mean that one band, often the lowest, is chosen as the baseline, and each of the other ten bands receives a coefficient describing how its outcome compares with that baseline. Few readers can hold ten contrasts in mind at once, and the cells at the extremes are often too small to produce stable estimates. Collapsing to three or four categories usually improves both precision and communication. The gain comes at a price, because everyone inside a category is treated as equivalent, so a respondent earning $31,000 and a respondent earning $89,000 become the same observation.
A threshold drawn from a poverty measure, a clinical guideline, or a published national category is defensible, because it refers to something outside the dataset and because other studies use it, which makes the estimate comparable. A threshold chosen because it produced the largest association is indefensible: the procedure guarantees an apparent effect, inflates the Type I error rate, and rarely replicates. The reason is that an analyst who tries many cut points and keeps the one with the strongest result is, in effect, running many tests and reporting only the most striking. Some cut point will show a large association by chance alone, even when income has no real effect, so the chance of a false-positive finding (a Type I error) rises well above the stated level. Because that chance pattern is specific to the sample, a new study using the same cut point usually fails to find it again. A median split sits between the two. It is convenient, but the threshold is a property of the sample rather than of the construct. By this we mean that the median income of one study's respondents may be $35,000 while the median in another study is $60,000, so “lower income” refers to quite different people in each, and two studies using median splits are not describing the same groups. The cut point and its rationale belong in the analysis plan, written before the data are examined.
Midpoint coding assigns $15,000 to everyone in the $10,000 to $19,999 band, which assumes respondents are distributed evenly across it. Income distributions are right-skewed, so the assumption holds imperfectly, and it fails entirely for the open-ended top band, which has no midpoint. A common convention assigns 1.5 times the lower bound, although any value chosen there is an assumption that should be stated in the methods. Coding the bands as consecutive integers carries a stronger assumption still, that the distance between adjacent bands is constant, which fails whenever the band widths differ.
A refusal is not a low income, and it is not a missed keystroke. Reserving distinct values, such as 8888 for a refusal and 7777 for “don’t know”, preserves information that matters, because refusal on an income item is patterned rather than random, and a concentration of refusals within one subgroup is a finding worth reporting.
Two cautions apply. Reserved values must be declared missing in the statistical software before analysis, because otherwise 8888 enters a model as a real observation and the error produces no warning. Reserved values must also lie outside the plausible range of the variable they occupy. The code 8888 is safely out of range for a banded variable numbered 1 to 11, but it is an entirely plausible income once the bands have been replaced by midpoints in dollars, so the same study may need different reserved values for the banded and the continuous versions of the same item.
A poverty threshold cannot be applied to personal income alone. It requires household size, and under most definitions geography as well, because the threshold differs between a single person in a small community and a family of four in a large city. An income-to-needs ratio requires the same companion items. If either appears in the analysis plan, the questionnaire must collect them, and the requirement becomes visible only when the analysis is worked through before the instrument is fielded. A coding scheme can be revised at any time. A companion item discovered to be missing after the field period has closed cannot be added.
Together these considerations imply that the coding scheme should be drafted alongside the questionnaire rather than after the data arrive. One practical test is to write the shell of the main results table before the instrument is finalised, with the rows and columns labelled as they will appear in the paper and the cells left empty. This forces every variable in the analysis to be traced back to an item on the questionnaire, and it exposes both the items that no analysis will use and the analyses that no item can support. Every coding decision that results, including reserved values, cut points, and derived variables, belongs in a codebook written and circulated with the questionnaire, so that the coder, the analyst, and any later user of the data read the same numbers in the same way.
A codebook fixes what each number means. The last step is to get those numbers into the computer without damaging them.
7.3 Data Entry and Management
Coding and data entry are separate jobs, and keeping them apart prevents errors that are very hard to find later. Responses should be coded directly on the paper questionnaire or data-capture form, using a distinctive ink colour so that the coder’s marks are easy to distinguish from the respondent’s answers, and only then entered. Combining coding and data entry into a single step increases the risk of errors. This is because the person doing both at once must read the respondent's answer, decide which code it deserves and type that code, all in one motion, and a slip at any of these points leaves no trace. When the codes are first written on the form, each decision is recorded on paper, so a later check can compare what was entered in the computer against what the coder wrote and find the mistake.
Computer data entry can be accomplished using general-purpose software such as spreadsheets or using specialised database managers. Each approach has advantages, and Table 3.3 compares the two on five features: ease of setup, validation rules, error prevention, the risk to data integrity and the transfer of data to statistical software.
Table 3.3. Comparison of spreadsheets and specialised database software for data entry across five features.
| Feature | Spreadsheets | Specialised Software |
|---|---|---|
| Ease of Setup | Easy | Moderate |
| Validation Rules | Limited | Extensive |
| Error Prevention | Low | High |
| Data Integrity Risk | High (column sorting can destroy records) | Low |
| Transfer to Statistics | Manual | Built-in |
Spreadsheets are easy to set up, but Table 3.3 shows their weakness: limited validation rules, little error prevention and a high risk to data integrity. The most serious of these risks is worth remembering on its own.
Never Sort a Single Spreadsheet Column
Use spreadsheets with caution for data entry. Sorting individual columns in a spreadsheet can completely destroy the data by separating a respondent’s answers across different rows. In a data spreadsheet each row holds one respondent, so the only thing linking a person's age to their smoking status is that the two values sit on the same row. If one column, such as age, is sorted on its own, its values move to new rows while every other column stays put, and each age is now paired with somebody else's answers. Nothing on the screen looks wrong, which is why the damage is so dangerous. Specialised database managers or statistical software with built-in data-entry modules are generally safer choices for larger studies.
This section has set out strategies for maximising response rates and the principles for coding responses, deriving analysis variables and entering data. The key takeaways below summarise these points.
Key Takeaways
- 7.1 A low response rate is more than an inconvenience: the people who do not respond often differ from those who do in their health, behaviour or attitudes, and when they do, your estimates are at risk of selection bias. (Strictly, the response rate is a proportion, the share of sampled people who complete the questionnaire, although everyone calls it a rate.) Strategies to raise response fall into two groups: design strategies, such as a clear purpose, a professional layout, thorough pre-testing, a short questionnaire and no unnecessary sensitive questions; and administrative strategies, such as small incentives, a stamped first-class return envelope, advance notice, a personalised cover letter and hand delivery. Following up non-responders with reminders and a fresh copy of the questionnaire is one of the most effective strategies, and small monetary incentives and recorded delivery also raise response substantially. When a scenario such as Worked Example 3.4 describes a low return, recommend changes from both groups, include follow-up of non-responders, and explain why it matters by linking low response to selection bias.
- 7.2 A coding plan decides, before any data arrive, how every possible answer will be recorded as a number. Give missing answers a single unique code that can never be a real answer, such as -999, and never leave a field blank, because a blank cannot be told apart from an item the coder skipped; never code a missing answer as 0 either, because 0 can be a real value and would be averaged in with the others. Give “don’t know” and “prefer not to answer” (a refusal) separate codes of their own too, such as 7777 and 8888 for the income item in Table 3.2, because a refusal, a genuine “don’t know” and a skipped item mean different things, and refusals are often patterned rather than random. Use the same conventions throughout the questionnaire, such as 0 = no and 1 = yes for every yes/no item, and remember that reserved codes must be declared missing in the statistical software before analysis so that they are not treated as real values. When you write a coding plan, check that every response option, including the non-substantive ones, has a code, and that no reserved code falls inside the plausible range of the variable.
- 7.3 Turning completed forms into a dataset is a separate job from coding them, and keeping the two steps apart prevents errors that are very hard to find later. Code the responses directly on the paper form, using a distinctive ink colour so the coder’s marks stand out from the respondent’s answers, and only then enter the data; combining coding and entry into one step increases the risk of mistakes. Spreadsheets are easy to set up but risky, because sorting a single column separates a respondent’s answers across different rows and silently destroys the dataset. Specialised database software, or statistical software with a data-entry module, is safer for all but the smallest studies, because it offers validation rules, stronger error prevention and built-in transfer to statistical analysis. If a question asks how to manage data entry, recommend coding on the form as a separate step and specialised software, and be ready to explain exactly how sorting one column can ruin a spreadsheet dataset.
1. A health authority emails a 30-minute questionnaire on physical activity to 3,000 adults. After 4 weeks, 750 (25%) have replied, and most of the replies come from people who say they exercise regularly. What should the team do?
2. For the item “In the past 30 days, on how many days did you smoke a cigarette? ____ days” (possible answers 0 to 30), a research assistant proposes this coding plan: record the number of days; leave skipped items blank; code “Don’t know” as 0; code “Prefer not to answer” as 0. Which revised plan follows the lesson?
3. A student enters 400 completed questionnaires into a spreadsheet, one row per respondent. To find the oldest respondents, she selects only the age column and sorts it from highest to lowest. What has happened to the dataset?
4. A public-health unit mails a questionnaire on household food security to a random sample of 1,500 households. After four weeks, 390 completed questionnaires have come back. The team then mails a reminder and a fresh copy to every household that has not replied, and a further 345 are returned. What are the response rates before and after the follow-up?
5. In the same food-security study, suppose that food-insecure households are less likely than other households to return the questionnaire. What is the likely effect on the estimate of food insecurity?
6. The food-security questionnaire includes the item “In the past 12 months, did anyone in your household eat less than they felt they should because there was not enough money for food?” (Yes / No / Don’t know / Prefer not to answer). Every other yes/no item is coded 0 = no and 1 = yes. Which coding plan is best?
✦ Pass the knowledge check with 100% to continue
Final Review & Assessment
⏱ Estimated time: 20 minutes
Bringing It All Together
This lesson walked through the full lifecycle of a research instrument: planning, drafting items, refining wording and structure, pre-testing, fielding, and finally coding the responses for analysis. Each stage compounds: a poorly defined objective produces vague items, vague items produce unstable measurements, and unstable measurements produce associations that look like findings but are really just noise.
The threads pulled together here also matter beyond questionnaire design. Cognitive theory of survey response (Tourangeau & Yan, 2007), validation, and pilot testing show up again whenever epidemiologists rely on self-reported exposures or outcomes, and the total survey error framework (Groves & Lyberg, 2010) ties these threads together. As you head into the final assessment, treat the takeaways below as the practical companion to the logic of sampling: you now know who to recruit and how to ask them what you need to know.
Key Takeaways from this lesson
- You can now explain that a questionnaire is the instrument and a survey is the study that uses it, and you can describe how a questionnaire is planned: define the objectives, consult interest holders (for example with the Delphi technique), run focus groups of 6 to 12 people from the study population and review existing validated instruments. For the exam, remember that planning comes before a single item is written, so give the steps in order and say what each one contributes.
- You can compare in-person, telephone, mailed and internet administration on response rate, cost, interviewer bias, reach and the handling of sensitive questions, and choose a method for a described study. No method wins on every count, so work through who the respondents are, whether they can read the form, how sensitive the topic is and what the budget allows, and justify your choice on the features that matter most for that study.
- You can describe the four cognitive steps in answering a question (understanding, retrieval, judgement and communication) and say at which step a flawed item is likely to fail. When you diagnose an item, read it as a respondent would and stop at the first step where an honest person would struggle; a daily count over a month or a year, for example, fails at retrieval, and the remedy is a shorter reference period or a typical day in a recent week.
- You can choose an appropriate question format, open or closed, and among checklist, multiple-choice, rating and ranking questions, applying the design rule for each type. Ask for exact numbers such as age wherever you can, because continuous data can always be grouped later but a range can never be turned back into an exact value; income is the usual exception and is asked in non-overlapping ranges with “Don’t know” and “Prefer not to answer”.
- You can name the rule a draft item breaks (too long or full of jargon, not specific about the event or time frame, double-barrelled, leading, or with categories that overlap or leave gaps) and write an improved version. On the exam, name the specific rule rather than calling an item “unclear”, check that your rewrite removes that fault without introducing a new one, and test the boundary values to spot overlapping or incomplete categories.
- You can lay out a questionnaire with an introduction, easy opening items, a funnel from general to specific within each section, sensitive items near the end and pre-coded responses, and you can explain what expert review, field pre-testing, think-aloud interviews, test-retest reliability and validation against a gold standard each reveal. Keep the distinction clear: pre-testing checks that the questionnaire works for real respondents, while validation checks that it measures what it claims to measure.
- You can explain why low response threatens validity, recommend strategies to raise it, and write a coding plan that keeps coding separate from data entry. Include follow-up of non-responders in any list of strategies and link low response to selection bias, and in a coding plan give missing answers, “don’t know” and refusals distinct codes, never a blank and never 0.
Final Knowledge Assessment
This assessment covers all material from this lesson. You must score 100% to complete the lesson. Review the feedback for any incorrect answers and try again.
1. A team developing a questionnaire on heat-related illness among older adults has two planning needs: (1) agreement among public-health officials, family physicians and housing advocates on the ten outcomes the questionnaire must measure, and (2) the words that older adults themselves use to describe symptoms of heat illness. Which pairing of planning tools fits these needs?
2. In a pilot study of vaping in two similar secondary schools, students in one school were asked the items in person by an interviewer, and 8% reported vaping in the past 30 days. Students in the other school completed the same items privately on a web form, and 19% reported vaping. Using Table 3.1, what is the most likely explanation for the difference?
3. A team will study diarrheal illness among farm-worker families living in seasonal camps. Many adults have limited reading ability in English, few have internet access or a reliable telephone, and the team can travel to the camps. Which administration method fits best?
4. A questionnaire for parents asks: “Has your child had an episode of otitis media in the past 6 months?” (Yes / No / Not sure). At which cognitive step will many parents struggle, and what is the fix?
5. A draft item reads: “Do you drink too much coffee? (Yes / No)”. At which cognitive step is it most likely to fail, and what is the fix?
6. A patient-satisfaction item reads: “How satisfied were you with your visit? ☐ Satisfied ☐ Dissatisfied”. The team plans to report the mean satisfaction score. Which revision follows the lesson’s rules for rating questions?
7. A draft item reads: “How long have you lived at your current address? ☐ Less than 1 year ☐ 1 to 5 years ☐ 5 to 10 years ☐ More than 10 years”. Which respondent cannot be placed in exactly one category, and what is the simplest repair?
8. Which of these draft items best follows the lesson’s guidance on collecting numbers?
9. A draft item reads: “Given the well-known dangers of second-hand smoke, do you support a ban on smoking in cars carrying children?” Which rule of question design does it break?
10. A draft item reads: “Have you recently experienced any adverse events following immunisation?” It breaks two wording rules: it uses jargon, and it is not specific. Which rewrite removes both faults?
11. A study asks: “How many migraines did you have in the past 30 days? ____”. In pre-testing, one respondent counts a headache that lasted three days as one migraine, while another counts it as three. Which rule of question design does the item break, and what is the fix?
12. When the same 50 adults complete a food frequency questionnaire twice, two weeks apart, their answers agree very closely. A colleague concludes: “So the questionnaire is valid: it measures true intake.” What is the best response?
13. Of 1,000 people who opened an online health survey, 300 (30%) closed it on the first screen, which had no introduction and asked for their exact household income. Which change follows the lesson?
14. To save time, a research assistant reads each paper questionnaire and types a code for each answer straight into the database, deciding on the codes as she goes. What is the main problem with this procedure?
15. A mailed questionnaire on chronic pain is sent to 1,500 adults, and 450 return it. Which statement gives the correct response rate and the correct interpretation?
16. A city health department plans a questionnaire to estimate how many residents aged 18 to 29 drove a motor vehicle within 2 hours of using cannabis in the past 30 days. It wants about 600 completed questionnaires; a similar survey suggests that about 30% of those invited will respond, and almost all young adults in the city use the internet. Which plan is best supported?
17. A first draft of the same questionnaire asks: “How often do you drive after using cannabis or drinking alcohol? ☐ Never ☐ Sometimes ☐ Often”. Which statement correctly diagnoses and repairs the item?
✦ Pass every section knowledge check with 100% before submitting