# Lesson 8: Collecting Quantitative Data: Surveys, Administrative Data and Chart Reviews

*Companion-podcast transcript, Sarah and Kiffer*

---

**Sarah:** Welcome back to Office Hours. I'm Sarah.

**Kiffer:** And I'm Kiffer. This is the episode for Lesson eight of Health Sciences two oh seven, Research Methods in Health Sciences. Last week we decided whom to study and what to measure. This week we collect the numbers.

**Sarah:** The lesson covers surveys, administrative data and chart reviews. Why put those three together?

**Kiffer:** Because they are the three main ways quantitative health data come into existence. You can ask people, you can reuse records the health system already keeps, or you can read clinical charts and copy out specific details. Each one measures things the others cannot, and each one fails in its own way.

**Sarah:** And our running case is still the Cedar Valley Social Connection Study.

**Kiffer:** Yes, which is fictional. Dr. Maya Hart's team is studying loneliness among adults aged sixty-five and older in a mostly rural health region in British Columbia. This week the team runs a regional survey that gets one thousand six hundred completed responses, links those responses to health records with consent, and reviews three hundred medical record charts at six partner clinics.

**Sarah:** Let's start with the survey. Most students picture a survey as a list of questions. What does this lesson add?

**Kiffer:** Everything that happens around the questions. Writing good questions is taught in Health Sciences three forty-one, Lesson three. Here the questions arrive already written, often from validated instruments like the three-item UCLA Loneliness Scale, UCLA being the University of California, Los Angeles. Our job is to put them on a platform, make the survey behave correctly, test it, send it out, follow up and export clean data. The lesson names six stages: build, test, pilot, distribute, monitor and export. Each one produces something you can check before moving on.

**Sarah:** Which platform should a student use?

**Kiffer:** The two you will meet most in Canadian academic health research are REDCap, which stands for Research Electronic Data Capture, and Qualtrics. REDCap was developed at Vanderbilt University for research, and member institutions run it on their own servers without a licence fee. Qualtrics is a commercial platform that many universities license. Either can handle a student project.

**Sarah:** So how do you choose between them?

**Kiffer:** Usually by what your institution provides and where the data will be stored. An ethics board will ask about the storage location, and so will partners. The team chose REDCap because the university hosts it, because one project could hold both the survey and the chart abstraction form, and because the partners accepted the arrangement.

**Sarah:** The reading says that building a survey is also building a dataset.

**Kiffer:** That is the idea I most want students to keep from the first section. Every question becomes a field, and every field becomes a column in the file an analyst will open later. The choices you make while clicking through the platform decide how easy or painful the analysis will be.

**Sarah:** What choices, specifically?

**Kiffer:** The field type, the variable name and the codes. The field type decides how the question is answered, for example one answer from a list or a typed number. The variable name becomes the column heading, something short and lower case with no spaces, like ucla one. The codes are the numbers stored for each answer, so the UCLA items are coded one for hardly ever, two for some of the time and three for often, because that is how the scale is scored.

**Sarah:** Then branching logic.

**Kiffer:** Branching logic is a rule that shows a question only to the people it applies to. The Cedar Valley survey asks whether you live alone, and only people who answer no are then asked who they live with. In REDCap you write the rule on the field that should appear, and in Qualtrics you use display logic or skip logic.

**Sarah:** The reading makes a point about blanks there.

**Kiffer:** It does, and students often miss it. After branching, a blank in the who-do-you-live-with column can mean two different things. For someone who lives alone it means the question did not apply. For someone who lives with others it means they skipped it. The only way an analyst can tell those apart is by knowing the rule, which is one reason the data dictionary records every rule.

**Sarah:** Remind listeners what a data dictionary is.

**Kiffer:** It is a table describing every field: its name, label, type, codes, validation and branching logic. REDCap stores the whole survey design as a data dictionary you can download as a spreadsheet. Anyone who analyzes the data later uses it to read every column correctly.

**Sarah:** What about validation?

**Kiffer:** Field validation is a check the platform applies as an answer is typed. The Cedar Valley age field accepts only whole numbers from sixty-five to one hundred and ten, so a typing error like seven is caught at the moment it happens, instead of weeks later during cleaning.

**Sarah:** It is tempting to make every question required so there is no missing data.

**Kiffer:** It is tempting, and an ethics board will usually push back. Under the Tri-Council Policy Statement, which everyone calls TCPS two, participants may decline to answer any question. The common approach is to require answers only on eligibility and consent, and to offer a prefer-not-to-answer option on sensitive questions. Then you still learn that someone chose not to answer, which is different from skipping by accident.

**Sarah:** Testing and piloting. Are those the same thing?

**Kiffer:** They answer different questions. Testing is the team working through the survey with test records to confirm the build does what it was designed to do. You enter a pretend respondent who lives alone, one who lives with a spouse, one who skips everything optional, and you check each path and the export.

**Sarah:** And piloting?

**Kiffer:** Piloting is a small number of people who resemble your participants completing the survey in realistic conditions. Cedar Valley used its advisory group of six older adults and twenty further pilot participants. They found the text was too small on some phones, a problem the team had missed on its office screens. The pilot also gave a median completion time of fourteen minutes, which comes back in Section two.

**Sarah:** Now distribution. The lesson distinguishes individual links from a public link.

**Kiffer:** An individual link is a web address unique to one invited person. A public link is one address anyone can open. Individual links let you know who has responded, remind only the people who have not, and stop one person answering twice. A public link is easy to share in a newsletter, and you give up control over who answers.

**Sarah:** How did Cedar Valley reach people?

**Kiffer:** The partner clinics mailed invitations to the random sample of five thousand two hundred and fifty older adults drawn in Lesson seven, and the team adapted Don Dillman's Tailored Design Method, which uses several planned contacts, each with a different job. There was a pre-notice letter from the health authority, an invitation with a web address and a unique access code, and a reminder postcard to everyone. Later, people who had not responded received a paper questionnaire, and finally a call or letter. Offering paper made the survey mixed-mode, which raised participation in the smaller communities.

**Sarah:** What does the evidence say about reminders and incentives?

**Kiffer:** A Cochrane review led by Phil Edwards found that follow-up contacts, shorter questionnaires and monetary incentives were among the strategies associated with higher response. On incentives, reviews including one by Eleanor Singer and Cong Ye have found that prepaid incentives, sent to everyone with the invitation, generally raise response more than promised incentives given only to people who finish.

**Sarah:** And yet Cedar Valley chose a promised incentive.

**Kiffer:** Because of the budget. A prepaid five dollar card to all five thousand two hundred and fifty people would have cost twenty-six thousand two hundred and fifty dollars. A promised ten dollar grocery card for an expected one thousand six hundred completers came to sixteen thousand. The team chose the promised card and noted in the protocol that the evidence favours prepaid incentives. To send the cards without attaching names to answers, the survey ends by sending people to a separate form for a name and address, stored apart from the survey data.

**Sarah:** Last part of Section one: tracking and response rates.

**Kiffer:** From the first mailing, the team logs every invitation. Of the five thousand two hundred and fifty selected, three hundred and five were unreachable or ineligible, which leaves four thousand nine hundred and forty-five eligible people. One thousand six hundred completed the survey, so the response rate is thirty-two point four percent. Counting the sixty partial responses gives thirty-three point six percent, and the simpler rate from Lesson seven, which divides by delivered invitations, gives thirty-one point seven. A report should state which one it used.

**Sarah:** Section two is about data quality. Where does it start?

**Kiffer:** With mode effects. A mode is the way the questions reach a person: web, paper, telephone or in person. A mode effect is a difference in results caused by collecting data in different modes, and it has two sources. A selection effect comes from different kinds of people choosing different modes. A measurement effect comes from the mode changing how the same person answers.

**Sarah:** Give us the Cedar Valley numbers.

**Kiffer:** Among one thousand and forty web respondents, twenty-one point zero percent scored six or higher on the loneliness scale. Among five hundred and sixty paper respondents, it was thirty-one point one percent. Overall it was twenty-four point five percent, which is three hundred and ninety-two lonely respondents out of one thousand six hundred.

**Sarah:** That is a big gap. Does paper make people lonelier?

**Kiffer:** The most likely main reason is who chose paper. Paper respondents were older and more likely to live alone in the smaller communities, and those characteristics go with loneliness. Both modes were self-completed, so there was no interviewer. The classic measurement effect involves interviewers, through social desirability bias, where people give the answer they expect the interviewer to view favourably.

**Sarah:** So what did the team do about it?

**Kiffer:** It kept the wording, order and response options identical on paper and on the web. It stored the mode as a variable so later analyses can compare or adjust for it. And it reported the split in the methods section so readers can judge for themselves.

**Sarah:** Then someone has to type in five hundred and sixty paper forms.

**Kiffer:** And every keystroke is a chance for error. In double data entry, two people enter every form independently and a program lists the differences. In verification, a second person re-enters a random sample. Cedar Valley used verification: a colleague re-entered fifty-six forms of forty fields each, which is two thousand two hundred and forty fields, and found seven errors. That is about zero point three percent, under the protocol's threshold of zero point five percent for re-entering everything.

**Sarah:** Now the part I suspect students find most surprising, which is bots.

**Kiffer:** A bot is a computer program that fills in online surveys automatically. A fraudulent respondent is a person who answers when they are not eligible, or answers many times, usually to collect a payment. Both concentrate in surveys that have a public link, offer an incentive and are advertised online. Researchers who recruited through social media, such as the team led by Pozzar, have described large numbers of suspicious responses arriving within hours of posting a link.

**Sarah:** Can't you just delete the bad ones afterwards?

**Kiffer:** If you can find them. The danger is the ones you do not find. They distort your estimates, use up the incentive budget, and in the worst case lead to findings about people who do not exist. Teitcher and colleagues have written about this, including the ethical trade-offs of screening people out.

**Sarah:** So what is the first line of defence?

**Kiffer:** Prevention, built in before launch. Individual access codes are the strongest protection, which is why the main Cedar Valley survey had very few problems. For an open link, a CAPTCHA, the short test that is easy for people and hard for many programs, blocks simple bots. You can describe the incentive in the consent form instead of advertising the amount online. And you can add a honeypot question.

**Sarah:** Explain the honeypot.

**Kiffer:** It is a field hidden from human respondents through the survey's code or styling. People never see it, so they leave it blank. Many bots fill in every field they find, so an answer in the honeypot is a strong sign of automation.

**Sarah:** And after launch, detection.

**Kiffer:** The team decides in advance which warning signs, or flags, it will look for. Cedar Valley used a completion time under one third of the fourteen minute pilot median, which works out to four minutes and forty seconds. It also used a failed attention check, an answer in the honeypot, a repeated email or internet address, inconsistent answers such as a postal code outside the region, and open text that is nonsense or copied.

**Sarah:** Why must the rule be written in advance?

**Kiffer:** Because otherwise you are tempted to exclude responses because you dislike what they say. Writing it first also lets the ethics board review it, and the consent form can tell participants that responses are screened for quality.

**Sarah:** Walk us through the worked example.

**Kiffer:** In the lesson's illustration, the team tested an open web link with a ten dollar e-gift card to reach people who had recently moved to the region. Over one weekend it received two hundred and twelve responses. Under the rule, a honeypot answer means exclusion, two or more flags mean exclusion, one flag means review by two people, and no flags means the response is kept. In the end, sixty-seven responses were retained and one hundred and forty-five excluded. The team then closed the link and switched to access codes handed out by community organizations.

**Sarah:** Why bother with the review step?

**Kiffer:** Because of cases like response R thirty-five in the reading, which had one flag, a postal code outside Cedar Valley. That person had just moved to Riverside and gave the old postal code, and everything else was consistent, so the reviewers kept it. Excluding every single flag would have thrown away exactly the recently moved older adults the link was meant to reach.

**Sarah:** Section two ends with security and export.

**Kiffer:** Security is mostly ordinary safeguards applied consistently. Store the data where your ethics approval says. Give each person only the access they need, so in Cedar Valley the community research associate can enter paper forms but cannot export. Keep names and addresses in a separate form. Use two-factor authentication, rely on the audit trail that records who viewed, changed or exported data, and never email exported files.

**Sarah:** And the export itself?

**Kiffer:** An export is a dated snapshot, usually of coded values together with the data dictionary. REDCap can remove fields tagged as identifiers and strip free text. Then you check the number of rows, whether every code is in the dictionary, whether the loneliness total recomputes correctly, and whether any test records crept in. If you find an error, you fix it in the platform and export again, so the platform stays the single source of the data.

**Sarah:** Section three moves away from asking people. What are administrative health data?

**Kiffer:** Records the health system creates to pay for services, manage care and register life events. When researchers reuse them, we call it secondary use, and the organization legally responsible for a dataset is its data custodian. In Canada these records are especially valuable because provincial insurance covers medically necessary physician and hospital care for nearly everyone.

**Sarah:** Why does Cedar Valley need them when it already has a survey?

**Kiffer:** Because the study's outcome is emergency department visits, and asking older adults to count a year of visits is unreliable. People forget, and they telescope, which means remembering events as more recent than they were. Records are created at the time of each visit. With consent, the team links survey responses to records of physician visits, hospital stays and emergency visits through Population Data BC. Lesson nine covers how that request and linkage work.

**Sarah:** The lesson describes a record as a by-product of a transaction.

**Kiffer:** That phrase explains most of what you need to know. A physician bills, a hospital reports a stay, a pharmacist dispenses, a death is registered, and the record holds what that transaction needed. In the reading's figure, one fictional participant's family doctor visit, emergency visit, hospital admission and filled prescription all leave records. Her weekly seniors' lunch, her loneliness and the pain reliever she bought over the counter leave nothing.

**Sarah:** Let's go through the four main record types.

**Kiffer:** Physician billing claims first. In British Columbia, a fee-for-service physician bills the Medical Services Plan for each visit, and the claim records the patient, date, physician, type of service and usually a single diagnostic code. Claims are good for counting physician contacts. They say nothing about what happened in the visit, and physicians paid by salary or contract may submit less complete records.

**Sarah:** Hospital discharge abstracts?

**Kiffer:** When someone leaves hospital after an admission, trained coders read the chart and code the stay, using the Canadian version of the tenth revision of the International Classification of Diseases. Outside Quebec, these go to the Discharge Abstract Database at the Canadian Institute for Health Information. They are the best source for hospital stays, with more diagnostic detail than billing claims.

**Sarah:** And prescriptions?

**Kiffer:** In British Columbia, PharmaNet records every prescription dispensed by a community pharmacy, whoever pays. It gives the drug, date, quantity and prescriber. It cannot tell you whether the person took the drug or why it was prescribed, and drugs given in hospital are missing.

**Sarah:** Vital statistics?

**Kiffer:** The registrations of births, deaths, stillbirths and marriages. Death registrations include the underlying cause, coded from the medical certificate of death, so they are excellent for whether and when someone died, and the cause is only as accurate as the certificate.

**Sarah:** The lesson suggests a habit for deciding which source to use.

**Kiffer:** List every variable your study needs and decide, one by one, which source measures it best. For Cedar Valley, loneliness and living alone come from the survey, because no administrative file records them. Emergency visits, hospital stays and deaths come from administrative records. Family physician visits come from billing claims, with the caveat about salaried physicians. And whether a clinician wrote down that someone seemed isolated comes from chart review.

**Sarah:** What is a case definition?

**Kiffer:** A rule that combines records to decide who has a condition. A single code can mislead, because a physician ordering a test to rule out diabetes might put a diabetes code on the claim. So researchers use definitions like the one Janet Hux and colleagues validated in Ontario: at least one hospital abstract with diabetes, or at least two physician claims with a diabetes code within two years. Codes also shift when coding systems, fee items or payment models change, so you read the custodian's documentation before interpreting a trend.

**Sarah:** So the survey and the records each cover the other's gaps.

**Kiffer:** That is the logic of the design. The survey gives the exposure, loneliness, which no record contains. The records give the outcome, emergency visits, without relying on memory.

**Sarah:** Section four is chart review. What exactly is an electronic medical record?

**Kiffer:** The digital chart a clinic keeps for each patient. A chart review is a method in which researchers read existing charts and copy specific, predefined information into a research form. That copying is called abstraction, the person doing it is an abstractor, and the form is the abstraction form.

**Sarah:** It sounds simple.

**Kiffer:** It can look simple, which has caused problems in published research. Gilbert and colleagues examined chart review studies in emergency medicine in nineteen ninety-six and found that most did not report basic features of their methods, such as how abstractors were trained, how variables were defined, or whether anyone checked the work. Later guidance from Vassar and Holzmann and from Kaji and colleagues repeats the same recommendations.

**Sarah:** What in a chart would a researcher want?

**Kiffer:** Charts hold structured data, such as the problem list and medication list, in fields with set formats. They also hold unstructured data, the free-text progress notes and scanned letters. For a social question like ours, the information is mostly in the notes. Whether a physician wrote that a patient seemed isolated, or referred them to a community program, lives in free text, and a person has to read it.

**Sarah:** Tell us about the Cedar Valley chart review.

**Kiffer:** The team reviews three hundred charts, fifty at each of six partner clinics, chosen by systematic sampling from each clinic's list of patients aged sixty-five and older, and used without individual consent under conditions the ethics board accepted. The review window is the twenty-four months before an index date set in the protocol, fixed in advance so that every abstractor reads the same period. Two abstractors, the graduate research assistant and the community research associate, work on site and enter coded values into a REDCap form.

**Sarah:** What does the form look like?

**Kiffer:** Each variable has a definition, a place to look and allowed values. Living situation has three values: lives alone, lives with others, and not documented. Loneliness noted by a clinician is yes if any progress note in the window uses a term from the manual's list, such as lonely, isolated or no one to talk to. Other fields cover referrals, visits, falls and depression on the problem list.

**Sarah:** The lesson calls the difference between no and not documented the most important idea in chart abstraction.

**Kiffer:** A chart that says nothing about living situation tells you only that nobody wrote it down. If abstractors record that silence as no, the study mistakes gaps in documentation for facts about patients. So every field needs a rule for what silence means, and that rule lives in the abstraction manual.

**Sarah:** How are abstractors trained?

**Kiffer:** They read the manual together and practise on charts with an answer key. They meet weekly to talk through difficult charts, and each decision is added to the manual with a date. And they are kept unaware of the study hypothesis.

**Sarah:** Why does that last part matter?

**Kiffer:** Imagine you expected lonely patients to visit the clinic often, and you open a busy chart with a note saying she has not been getting out much since winter. You might read that as isolation more readily than you would in a chart with two visits. So abstractors receive only the study number and the index date, and the hypothesis is not discussed with them.

**Sarah:** Then the agreement check.

**Kiffer:** Before main abstraction, both abstractors independently abstracted the same thirty charts, ten percent of the three hundred. For each variable they calculated percent agreement: the number of charts where both recorded the same value, divided by the number both abstracted, times one hundred. The target was ninety percent.

**Sarah:** And for the loneliness note variable?

**Kiffer:** They both recorded yes on six charts and both recorded no on eighteen, so they agreed on twenty-four of thirty, which is eighty percent. When they reviewed the six disagreements, five involved notes where a family member, rather than the patient, described the isolation. One abstractor counted those and the other did not. They added a rule to the manual, retested on twenty new charts and agreed on nineteen, which is ninety-five percent.

**Sarah:** So the check found a problem with the rule itself.

**Kiffer:** Usually that is what an agreement check finds, a definition that leaves room for interpretation. During main abstraction they also double-abstracted one chart in ten to catch drift, the slow change in how a person applies a rule over weeks of work.

**Sarah:** The lesson is careful to say that percent agreement has limits.

**Kiffer:** Two of them. First, agreement measures consistency, which is a different thing from accuracy. If both abstractors overlook referrals in scanned letters, they agree perfectly and both are wrong. Second, agreement can happen by chance when one answer is rare. If referrals appear in about five percent of charts and each abstractor marked yes on a random five percent without reading anything, they would still agree on about ninety point five percent of charts.

**Sarah:** So a high agreement figure can hide careless abstraction.

**Kiffer:** It can, and that is why a statistic called Cohen's kappa exists. Jacob Cohen introduced it in nineteen sixty to adjust agreement for the amount expected by chance. Kappa is taught in Health Sciences two forty-one, Lesson seven, and Health Sciences three forty-one, Lesson five. For now it is enough to know the name and the reason for it.

**Sarah:** Privacy must be a large part of chart review.

**Kiffer:** Clinic charts are among the most sensitive records in health care, and the clinic, as custodian, decides whether researchers see them. Cedar Valley abstractors enter only coded values and never copy notes word for word, because notes contain names.

**Sarah:** Let's bring the three sources together.

**Kiffer:** The survey measures what older adults report, and it depends on who responds and on memory. The administrative data measure what services recorded, without relying on memory, and they say nothing about loneliness. The chart review measures what clinicians wrote, and it depends on documentation habits. Together they answer a question none of them could answer alone.

**Sarah:** Which brings us to the practical side. What does a survey look like when it is ready to go out?

**Kiffer:** It has lower-case variable names, coded response values, branching rules and sensible validation, all built on the platform. The team enters at least three test records that follow different paths, exports them, and checks the export against the data dictionary.

**Sarah:** And the plan for running it?

**Kiffer:** A half-page operations plan states the distribution mode, contact schedule and reminders, any incentive and how it will be delivered, the fraud prevention measures and decision rule, and where the data will be stored and who can see them. That plan becomes the core of the study's data management plan. An interview study needs the operational points that still apply, such as scheduling, reminders, incentives and secure storage of recordings, which Lesson ten takes up. And if a study also uses records or charts, a short table lists each variable with its best source.

**Sarah:** Next week is Lesson nine, Data Sources and Data Linkage.

**Kiffer:** Where we pick up what this lesson deferred: how researchers apply for administrative data, how records from different files are linked, and how privacy is protected along the way.

**Sarah:** Thanks, Kiffer.

**Kiffer:** Thanks, Sarah. See you next week.