HSCI 312 · Lesson 11

Evaluation and Careers
in Health Promotion

Health Promotion: Individuals and Communities

Learning objectives for this lesson:

  1. Define evaluation in plain terms, and explain how the accountability movement and the move to evidence-based practice made it a standard part of the program environment in the United States and in Canada.
  2. Identify the four reasons to evaluate, and explain how a theory-based program tells the evaluator what ought to change and therefore what to measure.
  3. Distinguish process, outcome, and impact evaluation by the question each answers, the data each needs, and the time frame each requires, using fidelity, dosage, and baseline correctly.
  4. Build a basic logic model from the logical chain of the PRECEDE assessments, and identify the type of evaluation that tests each of its columns.
  5. Compare evaluation designs from record keeping to the classic experiment, identify the confounds that weaken a claim that a program caused a change, and explain why no single design suits every evaluation.
  6. Describe the career settings in which social and behavioural theory and evaluation are applied, in general and in Canadian public health, and how theory is used in each.
  7. Explain how theory and evaluation enter the funding process, from an agency's funding announcement to a community organization's grant application.

This course was developed by Dr. Kiffer G. Card, to accompany Edberg, M. Essentials of Health Behavior: Social and Behavioral Theory in Public Health. Jones & Bartlett Learning. This lesson follows Chapters 14 and 16 of the text.

Reference

Glossary: Key Terms, People & Concepts

📚 Reference page, available throughout the lesson

This glossary collects the key concepts, people, and ideas you will meet in this lesson. Use it as a reference while you work through the material, or as a review before assessments. Type in the search box to filter entries.

Core concepts
Evaluation In plain terms, the process of making sure you did what you proposed to do, determining whether the program had an effect based on what you were trying to achieve, and assessing whether the program model and the theory used to design it were useful. It may also cover cost-effectiveness and whether community goals were met.
Accountability The requirement to show that public or donor resources invested in a program had some effect. In the United States the Government Performance and Results Act of 1993 and the Program Assessment Rating Tool made performance monitoring a condition of funding; Canadian federal and provincial bodies apply comparable results policies.
Evidence-based practice The move within public health to build a body of interventions supported by evaluation evidence, in the same way that medicine developed evidence-based standards of care. Evaluation data are the evidence; registries compile and rate the programs that have it.
Model program A program that has demonstrated good evidence of effectiveness, especially one implemented in several places with different populations and found effective in each. Funders increasingly require grant recipients to use model or evidence-based programs, or to justify why none fits the community.
Fidelity The degree to which a program is implemented according to its design, with all components delivered as planned. Replicating a program with fidelity is what allows the replication to serve as an additional test of the model; process evaluation supplies fidelity data.
Dosage In a health promotion or other nonmedical intervention, the amount of the intervention (number of education sessions, number of activities, and so on) that results in a particular outcome or effect.
Baseline Data collected on the things a program expects to change before the program begins, so that the same information collected at one or more follow-up points can measure change. Also called pretest data; the later collection is posttest or follow-up data.
Inputs The resources, staff, program components, funds, and the like invested in or used for an intervention. Some of this information comes from the administrative and policy assessment of PRECEDE-PROCEED.
Outputs The activities a program delivers, such as educational materials, community events, health screenings, or trainings, together with the factors each activity is intended to change. The choice of activities is guided by the factors identified in assessment and by the theory judged appropriate.
Indicators and measures The criteria and tools for measuring outcome and impact: health data, survey or qualitative data, measures of skill, and the like. They are the last component of a logic model and they tie theory to observable evidence.
Confound An unplanned occurrence or other factor that may affect an intervention and complicate the claim that only the intervention caused a change. Seven common confounds are history, maturation, testing, regression to the mean, selection bias, mortality or attrition, and diffusion of treatment.
Independent variable A variable presumed to influence the dependent variable. In an evaluation the main independent variable is the intervention itself, that is, whether and how much of the program a person received. Characteristics the program is not trying to change, such as being a woman or a recent immigrant, are usually treated as covariates; they tell a great deal about who and what the program can change.
Dependent variable A characteristic of a person or situation that the intervention is trying to change, such as diet or exercise behaviour. It is what an evaluation of outcome or impact measures.
Quantitative data Information that can be expressed numerically and analyzed statistically, such as fixed-choice survey answers or official demographic and epidemiological records. Because it is collected the same way from many people, it supports comparisons and generalized conclusions about a population.
Qualitative data Narrative, descriptive, and subjective information from extended interviews, focus groups, and observation, usually gathered from fewer people. It reveals the details and meaning of events, how people interpret situations, and the context in which behaviour occurs.
Program officer A staff member of a government or public funding agency who helps decide the program framework behind a funding announcement and then manages or oversees the organizations selected to carry out the work, providing technical support and monitoring evaluation, without delivering the program directly.
Advocacy organization A nonprofit whose focus is increasing public engagement about an issue and affecting public policy on it, typically through media advocacy, social marketing, other communications efforts, and direct policy work such as position papers and draft legislation.
Types, models, and methods
Process evaluation Evaluation of how implementation fared. Its basic question is whether the components of the intervention were implemented as planned, answered by keeping records of materials, sessions, partners, and participation and comparing what was done with what was planned. It also supplies fidelity and dosage data.
Outcome evaluation Evaluation of the short-term or immediate effect of an intervention, measured by collecting baseline and follow-up data on the things theory and assessment predicted would change, such as knowledge, service use, or policy. In the convention this course uses, outcome refers to the short term; Green and Kreuter's PRECEDE-PROCEED framework uses the label the other way around.
Impact evaluation Evaluation of whether the intervention affected the health problem that was its ultimate target, for example cancer morbidity and mortality, measured with baseline and follow-up data over an extended period or by following a cohort of participants. In the convention this course uses, impact refers to the long term; Green and Kreuter's PRECEDE-PROCEED framework uses the label the other way around.
Formative evaluation Evaluation used during program development to shape the program before or as it is implemented, for example by pretesting materials or gathering feedback. This lesson names it, with cost-effectiveness and quality assurance evaluation, as a type it does not cover in detail.
Logic model A diagram or structure that links what a program plans to do with its expected outcomes and impacts. It connects the health problem, inputs, outputs, projected short-term outcomes and long-term impacts, and the indicators and measures used to evaluate them.
Quasi-experimental design An outcome and impact evaluation design that identifies a community or population similar to the intervention sample but not receiving the intervention, and collects pre- and posttest or time series data from both. Its problems are comparability, contamination of the comparison group, and other events in the intervention community.
Classic experimental design The most rigorous evaluation design: individuals in the target population are randomly assigned to the intervention or to serve as controls, and pre- and posttest or time series data are collected from both groups. Its problems include accidental exposure of controls and the ethics of withholding an intervention from a community that needs it.
Empowerment evaluation A collaborative approach, drawing on Paulo Freire's community theories and participatory programming, in which the community identifies its goals and the best ways to measure them, often instead of standard validated instruments. Its key purpose is to help a community improve its program on its own terms.
Guide to Community Preventive Services (Community Guide) A compendium from the United States Centers for Disease Control and Prevention that summarizes what is known about the effectiveness, economic efficiency, and feasibility of community interventions, based on systematic reviews, with recommendations from the Task Force on Community Preventive Services.
People
Paulo Freire Brazilian educator whose Pedagogy of the Oppressed (1970) argued that people should analyze and act on the conditions of their own lives. Empowerment evaluation traces its roots to his community theories and to participatory programming more broadly.
David Fetterman Evaluator whose Foundations of Empowerment Evaluation (2000) set out the approach described in this lesson: a collaborative effort with a community to define goals and measures, not an experimental design meant to produce rigorous comparative data.
Lawrence Green and Marshall Kreuter Developers of the PRECEDE-PROCEED planning framework, whose sequence of social, epidemiological, behavioural and environmental, educational and ecological, and administrative and policy assessments forms the logical chain that this lesson turns into a logic model.
No matching entries. Try a different search term.
Section 1

Evaluation: What It Is and Why It Matters

⏱ Estimated reading time: 15 minutes

Section 1 of 4

Evaluation: What It Is and Why It Matters

Did the whole process, from assessment to theory to program, make a difference?

Without the jargon

Evaluation is a short list of questions

  • Did we do what we proposed to do?
  • Did the program have an effect, judged against what we were trying to achieve?
  • Were the program model and the theory behind it useful?
  • Sometimes: what did it cost, and were community goals met?
The program environment

The accountability movement

1980s

Consensus that better evaluation was needed to improve planning and performance.

1993 onward

The Government Performance and Results Act, then the Program Assessment Rating Tool: monitor, report, account.

Planning documents

Stated goals become the measure of success. Healthy People in the United States; Treasury Board results policies in Canada.

Evidence-based practice

Evaluation data are the evidence

Evaluate

Without evaluation there is no evidence to enter into the record.

Registries rate

The Community Guide, NREPP, Health Evidence. Replication across settings counts most.

Funders require

Use a model program, or justify why none fits your community.

Why evaluate?

Four reasons

Accountability

Show that resources had an effect; meet reporting requirements.

Learning

See what is working and change course as you go.

Theory

Test a theoretical linkage or a model program in a new setting.

Efficiency

What did it cost to achieve the goals?

Theory and evaluation

Theory says what ought to change

Assessment identifies the factors. Theory predicts how changing them changes behaviour and health. Evaluation checks whether the predicted changes occurred.

The result is evidence about the program and, at the same time, evidence about the theory. A shortfall can mean the theory did not fit, or that the program was not delivered as designed.

Carry forward

What to take into the next section

  • Evaluation: did we do it, did it work, was the theory useful, what did it cost?
  • Accountability and evidence-based practice make evaluation a condition of funding.
  • Theory names the changes an evaluation must measure.
  • Next: process, outcome, and impact evaluation, and the questions each answers.

Overview

Health promotion planning follows a familiar path: assess a situation, identify the factors that matter, choose a theory that explains them, and design a program. This section asks the question that path leads toward. Did it work? This is not a course on evaluation, so the aim is familiarity rather than technique, because evaluation is so much a part of the program environment that anyone who plans programs will eventually be doing it. The section sets out what evaluation asks in plain terms, how the accountability movement and evidence-based practice made it a condition of funding, the four reasons for evaluating, and why a theory-based program is the kind that gives an evaluator something specific to measure.

Learning Objectives

  • 1.1 Define evaluation in plain terms as a short list of questions about a program.
  • 1.2 Explain how the accountability movement, in the United States and in Canada, made a program's stated goals the measure of its success.
  • 1.3 Describe what the move to evidence-based practice means for health promotion, including evidence registries and model programs.
  • 2.1 Identify the four reasons to evaluate and classify a situation by the reason that is driving it.
  • 2.2 Explain how a theory-based program tells the evaluator what ought to change, and how evaluation can separate a theory that did not fit from a program that was not delivered as designed.

1.1 Evaluation Without the Jargon

Evaluation sounds formal and technical, and students often picture statistics, control groups, and long reports. Stripped down, however, it is a short list of questions that anyone running a program should be able to answer. In plain terms, evaluation is the process of making sure you did what you proposed to do, determining whether the program had an effect based on what you were trying to achieve, and assessing whether the program model and the theory used to design it were useful. By this we mean that evaluation looks at a program from three angles in turn: first at what was delivered, then at what changed, and finally at whether the ideas behind the program held up. Sometimes it also means assessing cost-effectiveness, whether community goals were met, and other issues relevant to the program. The purpose of keeping the definition this plain is to show that evaluation is not a separate technical activity added at the end, but a set of questions that follow directly from what the program set out to do. The four cards below take these questions one at a time. Each card gives the question and an example, most of them drawn from a home-visiting program for families with newborns.

Did we do what
we proposed?

The first question is the simplest: making sure you did what you said you would do. A community organization that promised twelve peer-led workshops and a resource website has to be able to show that the workshops happened and the site went live. Example: a health unit in northern Ontario proposes home visits to every family with a newborn in three communities; the evaluation begins by counting the visits actually made.

Did it have
an effect?

The second question is whether the program produced the change it was trying to achieve. The answer depends on what you set out to achieve, which is why this lesson keeps returning to program goals. Example: if the goal was more parents choosing safe infant sleep positions, the evaluation asks whether that behaviour changed after the visits, compared with before.

Was the theory
useful?

The third question is the one that ties evaluation to the whole planning process: assessing whether the program model and the theory or theories used in its design were useful. A theory-based program predicts a chain of change. Evaluation checks each link. Example: the home-visit program assumed that modelling by a trusted nurse would raise parents’ confidence, and that confidence would change practice. Did confidence rise? Did practice follow?

What else
matters?

Sometimes evaluation also means assessing the cost-effectiveness of a program, whether community goals were met, or other issues relevant to the program. A program can change behaviour and still be too expensive to sustain, or can meet its funder’s targets while missing the goals the community cared about most.

Notice that the questions build on one another. The first can be answered from records alone, because it only asks whether the planned activities took place. The second needs more than records: it needs a measure of change, judged against the goal the program set for itself. The third goes further still, because it asks whether the chain of change that the theory predicted actually held. For example, in the home-visiting program, a count of visits answers the first question, a before-and-after comparison of infant sleep practices answers the second, and a check on whether parents' confidence rose before their practice changed answers the third. The third question is the one that ties evaluation to the rest of program planning. Evaluation is the way you carry your use of social and behavioural theory through to the point where you can determine whether the whole process, of assessing a situation, identifying factors to address, and using the assessment with relevant theory to plan and implement an intervention, made a difference. Quite apart from that intellectual reason, funders now insist on evaluation, and the next subsection explains why.

1.2 The Accountability Movement

In addition to the intellectual reason, there is also a pragmatic reason to evaluate. In the current and future environment of program funding there is a strong emphasis on evidence and accountability. By this we mean that whoever pays for a program now expects to be shown what the money achieved: to keep its funding, a program needs evidence that it is doing something about the problem it was meant to address. This imperative is relatively recent. Serious evaluation of health promotion and behavioural interventions did not become the norm until a consensus formed among researchers and practitioners in the 1980s that better evaluation was needed to improve program planning and performance.

In the United States the turning point was legislative. The Government Performance and Results Act (GPRA) of 1993, followed by the Program Assessment Rating Tool (PART) of the White House Office of Management and Budget, required agencies to develop performance monitoring and accountability procedures to ensure that public funds were well spent. The influence of this accountability movement has since been integrated into standard practice at every level, including private and global health promotion efforts. One consequence is the spread of strategic planning documents that set goals as a guide to action and as a method for evaluating progress. The purpose of writing a goal down in this way is that it does two jobs at once: it tells staff what to work toward, and it fixes in advance the yardstick against which the program will later be judged. For example, if the plan says "we will reduce the number of youth who smoke by 25%," then documented change in youth smoking, compared with that goal, becomes the measure of success. The Healthy People documents of the United States Department of Health and Human Services (Healthy People 2010, Healthy People 2020, and their successors) are large, comprehensive planning documents of this kind.

Canada arrived at the same place by a different route. Federal departments and agencies, including the Public Health Agency of Canada, work under Treasury Board results policies that require them to set expected results, report performance against them, and evaluate their programs on a regular cycle. Provincial bodies apply similar expectations to the organizations they fund: Ontario's public health standards, for example, expect boards of health to plan and evaluate programs using evidence, and regional health authorities in British Columbia report against provincially set performance measures. The Canadian Evaluation Society offers a Credentialed Evaluator designation, a sign of how much of a profession program evaluation has become. The practical effect for a health promotion program is the same on both sides of the border: a planning document states a goal, and the evaluation measures progress against it.

Accountability asks a program to show results against its own stated goals. A deeper change, described in the next subsection, asks what counts as evidence that a kind of program works at all.

1.3 Evidence-Based Practice and Model Programs

An even more profound change than accountability, arguably, is the move within public health to develop a body of evidence-based practice, in the same way that medicine developed evidence-based standards of care. By this we mean that the field has tried to build up a shared record of which kinds of program work, so that planners choose an approach because there is evidence for it rather than because it seems like a good idea. The difference from accountability is worth holding on to. Accountability asks whether one program met its own goals; evidence-based practice asks whether a type of program works across many places, so that the next community can choose it with confidence. For health promotion and prevention this shift has three consequences.

First, evaluating the program becomes very important, because evaluation data are the evidence. Without evaluations of individual programs there would be nothing to build the shared record from. Second, many agencies have developed directories of interventions that have evidence of effectiveness, and these directories often rate programs by the quality of their evidence. The purpose of rating is to tell planners how far they can trust a claim of success, and replication counts heavily in that judgement. For example, a program implemented in several places with different populations and found effective in all of them is rated more highly than a program tried once, even if it worked that one time, because the repeated success shows that the result was not a fluke of one setting. Third, programs with good evidence are called model programs or something similar, and public and even private funders increasingly require grant recipients to use a model program, or to justify why no appropriate program of that type exists for the community, population, or situation to be addressed. In practice this means that a community organization applying for funds must either show that it is delivering a recognized model program or explain why none fits its situation. Table 11.1 lists four registries and guides, two from the United States and two from Canada, and sets out who maintains each one and what it does.

Table 11.1. Registries and guides that support evidence-based practice in the United States and Canada, who maintains each, and what each does.

Registry or guideWho maintains itWhat it does
Guide to Community Preventive Services (the Community Guide)United States Centers for Disease Control and Prevention, with the Task Force on Community Preventive ServicesActs as a filter for a scientific literature that can be large, inconsistent, uneven in quality, and inaccessible. Systematic reviews summarize the effectiveness, economic efficiency, and feasibility of community interventions, and the Task Force issues recommendations.
National Registry of Evidence-Based Programs and Practices (NREPP)United States Substance Abuse and Mental Health Services AdministrationProvided descriptive information and peer-reviewed ratings of outcome-specific evidence for interventions that prevent or treat mental and substance use disorders. The registry in its original form was closed in 2018 and replaced by a smaller evidence resource.
Health EvidenceMcMaster University, Hamilton, OntarioA searchable, quality-rated collection of systematic reviews on the effectiveness of public health interventions, built for Canadian practitioners and decision makers.
National Collaborating Centre for Methods and ToolsOne of six National Collaborating Centres for Public Health funded by the Public Health Agency of Canada, hosted at McMaster UniversitySupports evidence-informed decision making in Canadian public health with methods, tools, and training for finding and appraising evidence.

The consequence for research is that a great deal of effort now goes into identifying model programs, which means building or locating the evidence base, and along with that, compiling evidence for the usefulness or inapplicability of the theoretical approaches used to structure those programs. In other words, the evidence base is about theories as well as programs. For example, every well-evaluated Health Belief Model program, whether it succeeds or fails, is also a data point about the Health Belief Model, because it shows whether the changes that model predicts actually occurred in that setting. The label "evidence-based" is easy to misread, and the callout below sets out what it does and does not require.

Common confusion: "evidence-based" does not mean "randomized"

"Evidence-based" does not mean a program was tested with a randomized trial. A registry rates the quality of evidence on a scale, and replication across settings and populations counts heavily. As Section 2 shows, a process evaluation that documents faithful implementation is part of the evidence too, because without it no one can tell whether a disappointing result reflects the theory or the delivery.

Accountability and evidence-based practice explain why evaluation now happens so often. They do not say what any particular evaluation is for, which is the subject of the next subsection.

2.1 Four Reasons to Evaluate

Four reasons to evaluate cover most real situations, and each one points the evaluation toward a different audience and a different kind of data. The first is accountability, which responds to the public need to show that the cost and resources invested in a program had some effect, and, since GPRA and PART in the United States and results policies in Canada, to legal and policy requirements for monitoring systems. By this we mean that accountability looks outward, to the funder or the public, and asks for proof that the investment did something. The second is learning and improvement. As a program runs, evaluation data show what is working and what is not, and this feedback lets you make changes as you go. In contrast to accountability, learning and improvement looks inward, to the people running the program, and its data are most useful while the program is still under way.

The third reason is theory, which means testing the validity of a theoretical linkage, or testing a model program or approach that has been shown to work somewhere else. For example, a team might ask whether raising perceived susceptibility really did lead to more screening, as the Health Belief Model predicts, or whether a program that worked in one province also works in another. The fourth reason is efficiency and other issues, which covers what it costs for the program to achieve its goals, and other kinds of assessment. Here the question is not only whether the program worked but whether it achieved its goals at a reasonable cost.

In practice one evaluation often serves several of these reasons at once. However, it helps to know which reason is driving a particular question, because that shapes what you collect. An evaluation driven by accountability needs evidence of effect that an outsider will accept, one driven by learning and improvement needs timely feedback, one driven by theory needs measures of the specific changes the theory predicted, and one driven by efficiency needs cost data alongside results.

Interactive 11.1 lets you practise telling the reasons apart. It presents eight situations, two for each reason. Click a situation, then click the reason that best fits it, and read the feedback, which explains the reasoning behind each placement.

📊 Interactive 11.1: Why is this evaluation happening?

Click a situation, then click the reason from the list of four that best fits it. The feedback explains the reasoning.
Accountability
Learning and improvement
Theory
Efficiency and other issues
Select a situation to begin.
0 of 8 placed

The situations in Interactive 11.1 carry cue words worth learning. A funder or a reporting requirement signals accountability; feedback used to adjust the program while it runs signals learning and improvement; a predicted linkage, or a model program tried in a new place, signals theory; and cost per result signals efficiency. One of those reasons, theory, deserves a closer look, because it connects evaluation to every earlier stage of program planning.

2.2 How Evaluation Relates to Theory

How does evaluation relate to theory? The answer runs through every stage of program planning. A theory-based program is a set of predictions. By this we mean that when planners choose a theory, they are committing themselves to a claim about how change will happen: if the program changes certain factors, then the behaviour should follow. For example, the Health Belief Model predicts that if people come to see themselves as susceptible to a serious condition, and see a course of action whose benefits outweigh its barriers, they will act. Social Cognitive Theory, in contrast, predicts that observing a credible model and building self-efficacy will change behaviour. Each prediction names something that ought to change, such as perceived susceptibility in the first case or self-efficacy in the second, and each of those things is something an evaluation can measure. Theory, in other words, tells the evaluator what to look for, and two programs aimed at the same behaviour but built on different theories will need to measure different things. Figure 11.1 shows the loop from assessment to theory to intervention to evaluation, and back again.

Assessment what is the problem, and why Theory what ought to change, and how Intervention activities aimed at those factors Evaluation did the predicted changes occur? evidence about the theory and about the program records, surveys, health data

Figure 11.1. The loop from assessment (what is the problem, and why) to theory (what ought to change, and how) to intervention (activities aimed at those factors) and to evaluation, which asks whether the predicted changes occurred, draws on records, surveys, and health data, and returns evidence about the theory and about the program to assessment.

The loop also explains why evaluation can be called the way to test theory. The evaluation returns evidence about the program and, at the same time, evidence about the theory, because the things it measures are the things the theory said would change. This double role matters most when a program falls short. A disappointing result could have two very different explanations. The first is that the theory did not fit this population: the program did what it was meant to do, but the predicted changes did not follow. The second is that the program was not delivered as designed, in which case the theory was never really put to the test. An evaluation that records what was delivered as well as what changed can tell these two explanations apart, and the conclusion a planner should draw is very different in each case. Section 2 returns to this distinction under the heading of process evaluation and fidelity. For now, hold on to the idea that a program without a theory has nothing specific to evaluate except whether it happened, and a theory without an evaluation is a prediction nobody checked.

Worked Example 11.1 applies this idea to a peer-led program on vaping and nicotine for grade 7 and 8 classes in Winnipeg, built on Social Cognitive Theory, whose provincial funder asks for evidence before renewing it. As you read it, watch for the difference between evidence that a program happened and evidence that it had an effect.

Worked Example 11.1: The Funder Asks for Evidence

The case. A community organization in Winnipeg has run a peer-led program for three years in which trained older teens deliver sessions on vaping and nicotine to grade 7 and 8 classes. The sessions are built on Social Cognitive Theory: the peers model refusal skills, students rehearse them, and the program aims to raise students' confidence that they can turn down a vape without losing face. The provincial funder's renewal letter asks two questions. Is the program evidence-based? And what evidence does the organization have that it is working? The director has attendance sheets, a stack of thank-you notes from teachers, and a vague memory of a similar program in Ontario that was rated well in a registry. She has no baseline data and no follow-up survey.

Step 1: Name the theory and what it predicts. Social Cognitive Theory predicts a chain. Observing credible peer models and rehearsing refusal skills should raise refusal self-efficacy, and higher refusal self-efficacy should lead to less vaping. Each link in that chain names something that ought to change.

Step 2: Identify the reasons to evaluate. The funder's letter reflects accountability: it wants evidence that public money produced an effect, and its question about whether the program is evidence-based comes from the move to evidence-based practice. If the director argued the case on her own terms, she would invoke the theory reason (did the Social Cognitive Theory prediction hold for these students?) and learning and improvement (what should change in the sessions?).

Step 3: Ask what the existing records can answer. Attendance sheets answer the first evaluation question, whether the program did what it proposed. Thank-you notes show that teachers liked the program, which is not a measure of change in students. Her memory of a well-rated Ontario program does not make her program evidence-based, because a registry rates the evidence for a specific program, and she would have to show that hers is that program, delivered as designed.

Step 4: Work out what she should have collected. To answer whether the program had an effect and whether the theory was useful, she needed to decide at the start what the theory said should change, measure refusal self-efficacy and vaping before the first session (baseline), and measure the same things again afterward (follow-up).

What this shows. A program without measures of what its theory predicted can prove that it happened, and nothing more. The common mistake to avoid is treating activity records and satisfaction as evidence of effect.

The director's problem is common, and it points to the rest of the lesson. Attendance sheets answer the first of the evaluation questions but not the second or third, and answering those requires a plan, made before the program starts, for what to measure and when. The next section names the different kinds of evaluation that such a plan can include and the question each one answers.

This section has set out the questions evaluation asks, the accountability movement and the move to evidence-based practice that made evaluation a standard part of the program environment, the four reasons for evaluating, and the way theory tells the evaluator what to look for. The key takeaways below draw these ideas together, and the knowledge check asks you to apply them to short Canadian cases.

HEAL connection: outcomes are not endpoints

The HEAL Framework for Wellbeing introduced in Lesson 1 holds that health and harms are not endpoints: poor health can push people further down the social hierarchy and feed back into the structures that produced it. Two consequences follow for evaluation. A program that acts mainly on psychosocial state and behaviour should not be judged as if it could close a gap produced upstream, and a small effect is not evidence that a problem is a matter of personal effort. The outcomes worth measuring also include upstream ones, such as policies adopted, services that now reach a group, and changes in material circumstance.

HEAL calls the process by which collective behaviour reshapes the sociopolitical context social construction. The career settings in Section 6 take part in that process through the programs they fund, deliver, evaluate, and study.

Key Takeaways

  • 1.1 Evaluation is less mysterious than it sounds, because it comes down to a short list of questions. Did we do what we proposed? Did the program have an effect, judged against what we were trying to achieve? Were the program model and the theory behind it useful? And sometimes, what did it cost, and were community goals met? When an exam question describes an evaluation activity, match it to the question it answers: counting sessions answers the first, measuring change against a goal answers the second, and checking whether a predicted link held answers the third. A common trap is to treat a program that ran smoothly as a program that worked, when those are different questions.
  • 1.2 Serious evaluation of health promotion is relatively recent; it became the norm only after a consensus in the 1980s that better evaluation was needed. In the United States the Government Performance and Results Act of 1993 and the Program Assessment Rating Tool required agencies to monitor performance and account for public funds, and in Canada, Treasury Board results policies for federal bodies such as the Public Health Agency of Canada, and provincial standards for the organizations they fund, do the same job. The practical result is the strategic planning document: if a plan says youth smoking will fall by 25%, documented change in youth smoking compared with that goal becomes the measure of success. If a question asks what such a document does in evaluation, the answer is that it sets the goal against which change is judged; it does not supply data or design the evaluation.
  • 1.3 The move to evidence-based practice, modelled on evidence-based standards of care in medicine, is arguably an even more profound change than accountability. It means three things: evaluation data are the evidence, registries such as the Community Guide and Health Evidence compile and rate interventions by the quality of their evidence, and funders increasingly require a model program or a justification for not using one. A model program is one with good evidence of effectiveness, and replication in several places with different populations earns a higher rating than a single success. Avoid equating "evidence-based" with "tested in a randomized trial"; registries rate evidence on a scale, and process data showing faithful delivery are part of the evidence.
  • 2.1 There are four reasons to evaluate, and spotting which one is driving a question tells you what to collect. Accountability shows a funder or the public that resources had an effect, or meets a reporting requirement. Learning and improvement uses data while the program runs to change what is not working. Theory tests a predicted linkage, or tests whether a model program that worked elsewhere works here, and efficiency and other issues asks what it cost to reach the goals. In a case question, look for who is asking and why: a funder's renewal condition is accountability, weekly feedback used to adjust sessions is learning and improvement, and cost per result is efficiency.
  • 2.2 A theory-based program is a set of predictions, and each prediction names something an evaluation can measure. The Health Belief Model predicts that perceived susceptibility, with benefits that outweigh barriers, leads to action; Social Cognitive Theory predicts that credible models and growing self-efficacy change behaviour. Measuring each link returns evidence about the program and about the theory at the same time, as Figure 11.1 shows, and lets the evaluator separate a theory that did not fit the population from a program that was not delivered as designed. Remember the Winnipeg program in Worked Example 11.1: without baseline and follow-up measures of what the theory predicted, a program can show that it happened, and nothing more.
Knowledge Check: this section

1. A regional health authority tells a community organization that its harm reduction outreach program will be renewed only if it can document that the funds produced a measurable effect. Which of the four reasons to evaluate does this request reflect?

Learning and improvement looks inward: the people running the program use feedback while it runs to change course. Here an outside funder is asking for proof of effect as a condition of renewal, which looks outward.
The theory reason tests a predicted linkage or tries a model program in a new setting. The health authority is not asking whether a theory held; it wants evidence that its money produced an effect.
Accountability shows a funder or the public that the resources invested had an effect. A health authority tying renewal to documented, measurable effect is the funder's renewal condition the lesson names as accountability.
Efficiency asks what it cost to reach the goals, such as cost per result. The request is about whether the funds produced an effect at all, not about the cost of each result.
Correct answer: Accountability. The information needed is the list of four reasons in 2.1. The health authority wants evidence that its funds produced an effect, which is the definition of accountability, and it ties that evidence to renewal, the kind of requirement that followed the accountability movement. Learning and improvement is tempting because the organization may also learn from the data, but that reason describes the organization using feedback to adjust the program as it runs, not a funder asking for proof. (Section 1, 2.1 Four Reasons to Evaluate; Objective 2.1)

2. Why is a program that has been implemented in several places with different populations, and found effective in each, rated more highly in an evidence registry than a program tried once with success?

This confuses evidence-based with randomized. Registries rate the quality of evidence on a scale, and the lesson's callout stresses that evidence-based does not require a randomized trial.
Registries rate programs by the quality of their evidence, and success repeated in several places with different populations shows the result was not a fluke of one setting, so replication earns the higher rating.
The lesson makes no such assumption about single trials. The issue is the strength of the evidence: one success is weaker evidence than success replicated across settings and populations.
Registries rate evidence of effectiveness, not delivery cost. Cost belongs to the efficiency reason for evaluating, and nothing in the lesson says multi-site programs are cheaper.
Correct answer: replication across situations is stronger evidence. The information needed is how registries rate programs, described in 1.3 and Table 11.1. Registries rate programs by the quality of their evidence, and a program shown to work in several places with different populations is rated above one that worked once, because repeated success across settings is stronger evidence of effectiveness. The randomized-trial option is the most tempting, but "evidence-based" does not require a randomized trial, as the callout in 1.3 points out. (Section 1, 1.3 Evidence-Based Practice and Model Programs; Objective 1.3)

3. A youth mental health program built on Social Cognitive Theory shows no change in help-seeking. Which statement best captures how theory and evaluation are linked in interpreting this result?

One disappointing result cannot refute a theory until you know the program was delivered as designed. The lesson says a shortfall can mean the theory did not fit or that delivery failed.
The lesson argues the opposite: theory tells the evaluator what ought to change. Measuring only the final behaviour cannot show which link in the predicted chain broke.
Social Cognitive Theory predicts that modelling raises self-efficacy and self-efficacy changes behaviour, so each link can be measured. That lets the evaluator separate a theory that did not fit from a program not delivered as designed.
Being evaluated does not make a program a model program. Model programs are those with good evidence of effectiveness, and this result shows no change in help-seeking.
Correct answer: the theory predicted intermediate changes, so each link can be checked. The information needed is the loop in Figure 11.1 and the discussion after it in 2.2. Social Cognitive Theory predicts that modelling raises self-efficacy and that self-efficacy changes behaviour, so the evaluator can measure each link and tell whether the theory did not fit or the program was not delivered as designed. Concluding that the theory is wrong is the tempting error: one disappointing result cannot refute a theory until you know the program was delivered faithfully. (Section 1, 2.2 How Evaluation Relates to Theory; Objective 2.2)

4. The accountability movement accelerated the use of strategic planning documents. In evaluation terms, what role does such a document play?

A strategic planning document sets goals as a guide to action and a method for judging progress. If it says youth smoking will fall by 25%, documented change against that goal becomes the measure of success.
A planning document states the goal; it does not supply data. Baseline data still have to be collected so that change can be judged against the stated goal.
Lists of programs with good evidence come from registries such as the Community Guide or Health Evidence, not from strategic planning documents, which state goals.
Randomization is a feature of the classic experimental design chosen for an evaluation. A planning document sets the goal; it does not design the evaluation.
Correct answer: it states the goals against which documented change is measured. The information needed is the account of strategic planning documents in 1.2. A plan that says youth smoking will fall by 25% turns that goal into the test: documented change in youth smoking, compared with the goal, is the measure of success. The other options confuse the plan with parts of the evaluation itself; a planning document does not collect baseline data, list model programs, or set randomization procedures. (Section 1, 1.2 The Accountability Movement; Objective 1.2)

5. A diabetes prevention program in a Toronto neighbourhood delivered every workshop it promised and met the participation target in its funding agreement. Community members say, however, that it never addressed the goal they cared about most, access to affordable fresh food. In plain terms, which part of evaluation does their concern belong to?

Delivering every workshop answers the first question, whether the program did what it proposed, and the program already passes it. The community's concern is about a goal the program never targeted.
The funder's participation target has already been met. The community is not disputing that effect; it is asking about its own goal, access to affordable fresh food.
Whether the model and theory were useful asks if the predicted chain of change held. The community's complaint is not about the theory but about which goals the program served.
The plain-terms definition adds that evaluation sometimes covers whether community goals were met. A program can do what it proposed and meet its funder's target while missing the goal the community cared about most.
Correct answer: other issues, such as whether community goals were met. The information needed is the plain-terms definition in 1.1, whose last set of questions covers cost-effectiveness, community goals, and other issues relevant to the program. The program has already shown that it did what it proposed and met its funder's target, so the community's complaint is about a different goal, its own. Choosing "did what it proposed" is the trap: delivering every workshop shows implementation, not that the program served the community's goals. (Section 1, 1.1 Evaluation Without the Jargon; Objective 1.1)

6. The board of a nonprofit asks its coordinator to "do an evaluation" of a sexual health program for newcomer youth before the next funding cycle, but nobody says why. Why should the coordinator first find out which reason to evaluate is driving the request?

One evaluation often serves several reasons, but the one driving the request shapes what to collect: evidence of effect for accountability, timely feedback for learning, measures of predicted links for theory, and cost data for efficiency.
The lesson says the opposite: in practice one evaluation often serves several reasons at once. The point of knowing the driving reason is to decide what data to collect.
Accountability needs more evidence, not none: evidence of effect that an outsider such as a funder will accept, judged against the program's stated goals.
The theory reason tests any predicted linkage, whether from the Health Belief Model, Social Cognitive Theory, or another theory, or tests a model program in a new setting.
Correct answer: because the reason shapes what is collected. The information needed is the point in 2.1 that one evaluation often serves several reasons, but the reason driving a question determines what to collect. Accountability calls for evidence of effect against the program's goals, learning and improvement for feedback while the program runs, theory for measures of each predicted link, and efficiency for cost per result. The option saying only one reason can apply is wrong, because a single evaluation often serves several reasons at once. (Section 1, 2.1 Four Reasons to Evaluate; Objective 2.1)

7. The same program is built on Social Cognitive Theory: peer role models, rehearsal of skills, and self-efficacy. To serve the theory reason, what should the coordinator measure at baseline and follow-up?

Session counts answer only whether the program happened, the first evaluation question. As the Winnipeg case in Worked Example 11.1 shows, they cannot test whether the theory's predicted changes occurred.
Satisfaction shows that people liked the program, like the Winnipeg teachers' thank-you notes, but it is not a measure of the changes Social Cognitive Theory predicts.
Social Cognitive Theory predicts that peer modelling and rehearsal raise self-efficacy and that self-efficacy changes behaviour. Measuring both at baseline and follow-up tests each link, which is what the theory reason requires.
Cancer incidence over ten years is a long-term impact, unrelated to this sexual health program's predicted chain, and far beyond a single funding cycle. The theory reason needs measures of the specific links the theory names.
Correct answer: self-efficacy and the behaviour it is predicted to change. The information needed is the idea in 2.2 that a theory names the links in a chain, and those links become the things to measure. Social Cognitive Theory predicts that modelling and rehearsal raise self-efficacy and that self-efficacy changes behaviour, so both must be measured before and after the program to test the theory. Session counts are tempting because they are easy to collect, but they show only that the program happened, which is the lesson of Worked Example 11.1. (Section 1, 2.2 How Evaluation Relates to Theory and Worked Example 11.1; Objective 2.2)

✦ Pass the knowledge check with 100% to continue

Section 2

Process, Outcome, and Impact Evaluation

⏱ Estimated reading time: 15 minutes

Section 2 of 4

Process, Outcome, and Impact Evaluation

Three questions, three kinds of data, three time frames.

Three questions

Process, outcome, impact

Process

Were the components of the program implemented as planned?

Outcome

What short-term or immediate effect did the program have?

Impact

How did the program affect the health problem that was the ultimate target?

Process

Did we do what we planned?

Fidelity

All components delivered according to the design. Without it, a replication tests nothing.

Dosage

The amount of the intervention (sessions, activities) that results in an effect.

Records of sessions, materials, partners, and participation separate a failed theory from a failed delivery.

Outcome

Short-term effects, measured before and after

Baseline, then one or more follow-ups, on the things theory said would change first.

  • Knowledge, attitudes, skills
  • Health service utilization
  • Policy or regulatory change
  • The reach of a three- or four-year program
Impact

The health problem itself

Awareness

Susceptibility, benefits, barriers: the short-term outcome.

Quitting

The behaviour change the theory predicts will follow.

Cancer

Morbidity and mortality, years later: the long-term impact.

A note on labels

Which word means long term?

This course

Outcome = short term. Impact = long term.

PRECEDE-PROCEED

Impact = immediate effects on factors and behaviour. Outcome = health and quality of life.

Same idea, swapped labels. Check definitions before comparing results.

Carry forward

What to take into the next section

  • Process: implemented as planned? Fidelity and dosage.
  • Outcome: short-term change in targeted factors, baseline to follow-up.
  • Impact: the health problem itself, over years or a cohort.
  • Next: the logic model that gives each type its place, then methods and confounds.

Overview

Section 1 described evaluation as a short list of questions. Look closely at any real program, though, and there are many things one could evaluate and several goals an evaluation can serve. The way to make this manageable is to start with three basic categories: process, outcome, and impact evaluation. A project may use one, two, or all three at once, because each answers a different question with different data on a different time frame. This section works through the three types, the vocabulary that comes with them (fidelity, dosage, baseline and follow-up), and an awkward fact about the labels themselves, using Canadian programs that need all three.

Learning Objectives

  • 3.1 State the question that process, outcome, and impact evaluation each answer, and name other common types of evaluation.
  • 3.2 Explain how process evaluation, through fidelity and dosage data, separates a failed theory from a failed delivery.
  • 3.3 Describe the baseline and follow-up logic of outcome and impact evaluation, the data each collects, and the time frame each requires.
  • 3.4 Explain how sources differ in the time frames they attach to the labels outcome and impact, and check a report's definitions before comparing its results with those of other reports.

3.1 Three Questions

The simplest way to hold the three types apart is by the question each one asks. Process evaluation asks whether the components of the intervention were implemented as planned. By this we mean that it looks at the program itself, at what was delivered, rather than at the people the program was meant to change. Outcome evaluation asks what short-term or immediate effect the intervention had, so it turns from the program to its participants and looks for early change. Impact evaluation asks how the intervention affected the health problem or issue that was its ultimate target, which is usually the slowest thing to change. For example, in a program on healthy eating, process evaluation would ask whether the classes were held, outcome evaluation whether participants' knowledge and eating habits changed soon afterward, and impact evaluation whether the health problem the program was built to address became less common over the following years. For outcome and impact there is an added clause: what do your logic model and theory tell you should have happened? This clause matters because outcome and impact evaluation cannot measure "change" in general; they measure the particular changes the program's theory predicted. Other types exist that this lesson names but does not cover in detail: formative evaluation, used during program development; cost-effectiveness evaluation; and quality assurance evaluation. Open each of the four tabs below. The first three give the question, the method, and the timing for process, outcome, and impact evaluation in turn, and the fourth describes the other types.

Process evaluation

Question: Were the components of the intervention implemented as planned?

How: keep records. How many materials were developed and distributed, how many education sessions were held, how many community coalition partners took part, how many people showed up. Then compare what you actually did with what you planned to do.

When: throughout implementation. The historical recordkeeping approach, which tracks project activities and client participation, is the basic process evaluation method.

Outcome evaluation

Question: What short-term or immediate effect did the intervention have?

How: collect data on the things you expect to change in the short term before the project begins, then collect the same information at one or, preferably, more points after the intervention is in place. This is pre- and posttest data, or baseline and follow-up data.

What: depending on program goals, knowledge change, changes in health service utilization, policy changes, or other educational, ecological, or policy data. It is the kind of change a program funded for three or four years could expect.

Impact evaluation

Question: Did the intervention affect the health problem or issue that was the ultimate target?

How: the same baseline and follow-up logic, but the data concern the long-term impacts you expect, such as cancer morbidity and mortality, and follow-up continues for years.

When: only if the project is in place for an extended time (more than three or four years) or if you can follow the people who took part, a cohort, over an extended period.

Types this lesson names but does not cover

Formative evaluation is used in program development, before or as a program is rolled out, to shape it: pretesting messages, piloting a session, gathering feedback from the intended audience. Cost-effectiveness evaluation asks what it costs to achieve a unit of result. Quality assurance evaluation checks that services meet defined standards. Qualitative approaches can be part of formative work as well as of process, outcome, and impact evaluation.

Read across the first three tabs and a pattern appears. The three types differ in what they ask, but outcome and impact share a method, collecting the same data before and after the program, and differ mainly in what they measure and how long they must wait. The next three subsections take the types in turn, beginning with the one that is easiest to underrate.

3.2 Process Evaluation: Did We Do What We Planned?

Process evaluation is answered by keeping records: how many materials were developed and distributed, how many education sessions were held, how many community coalition partners took part, how many people showed up. You then compare what you actually did with what you planned to do. By this we mean that the program plan serves as the standard, and the records show how closely the delivered program matched it. For example, if a plan promised ten cooking classes for fifty families and the records show six classes attended by twenty families, the process evaluation has found a gap between plan and delivery before anyone has measured a single change in diet. This historical record keeping, tracking project activities and client participation, is the basic process evaluation method. Because the records describe delivery as it happens, they have to be kept throughout implementation rather than pieced together at the end.

It is tempting to see process evaluation as mere bookkeeping. However, it is more than that, because it matters for testing theory. Recall from Section 1 that a disappointing result could mean either that the theory did not fit or that the program was not delivered as designed; process evaluation is what tells these apart. For example, suppose an intervention is based on the stages of change model from the Transtheoretical Model. Its effectiveness may depend in part on periodically assessing each participant to identify their stage in changing, say, their diet, because the program's components are tied to a stage for each individual. If the intervention turns out less effective than expected, it would be very helpful to know, through process evaluation, whether the stage assessments were done properly. If they were not, then at least you know that the problem may lie in the details of implementation rather than in the theory or the intervention design. Without that record, the planners might abandon a sound theory for a failure that was really one of delivery.

Two terms follow from this, and they are easy to confuse because both describe delivery. Fidelity is implementing all program components according to the program design. By this we mean that fidelity asks whether the program was delivered in the form it was designed to take, with every component in place. The purpose of checking it is that fidelity is what allows a replication of a program to count as an additional test of the model; a replication that dropped or changed components is testing something else. Dosage, in contrast, is the amount of an intervention, such as the number of education sessions or activities, that results in a particular outcome or effect. Dosage asks how much of the program people received. The two can come apart. For example, a program could be delivered with every component intact, and so with good fidelity, while many participants attend only a few sessions and so receive a low dose. The first two expand on fidelity and dosage, the third describes record keeping, and the fourth introduces the baseline and follow-up data on which outcome and impact evaluation depend, which 3.3 takes up.

Fidelity

A program that has shown evidence of effectiveness in one situation may be replicated in another. For that replication to serve as an additional test of the model, it has to be carried out with fidelity: all program components implemented according to the program design. Process evaluation provides the fidelity data. Without it, a replication that fails tells you nothing about the model, because you cannot say whether the model or the delivery failed.

Dosage

It may sound odd to use a medical term for a health promotion program, but dosage refers to the amount of an intervention, such as the number of education sessions or the number of activities, that results in a particular outcome or effect. A student who attended two of six sessions received a different dose than one who attended all six, and a process evaluation that records attendance lets the outcome evaluation take that into account.

Record
keeping

The basic process evaluation method is historical record keeping: track project activities and client participation. The data are counts of educational sessions conducted, health services delivered, calls taken, advertisements posted, client utilization, and so on. The question is whether program inputs are creating the planned outputs. Most of these records cost little to keep if the forms are designed before the program starts, and are almost impossible to reconstruct afterward.

Baseline and
follow-up

Outcome and impact evaluation depend on collecting data about the things you expect to change before the project begins (baseline, or pretest) and then collecting the same kind of information at one or preferably more points after the intervention is in place (follow-up, or posttest). The word "same" matters: a change can only be measured if the second measurement is comparable to the first.

Together the cards make one point: without records of what was delivered, to whom, and how much, a disappointing result cannot be interpreted. Worked Example 11.2 applies fidelity and dosage to a replication that went wrong. Watch for the conclusion the superintendent draws, and for what the process data say instead.

Worked Example 11.2: A Replication That Disappointed

The case. A school district in Alberta adopted a mental health literacy curriculum that had shown good results in another province. At follow-up, students' knowledge and attitudes had barely moved. The superintendent concluded that the curriculum did not work for Alberta students. The district's process evaluation, however, showed that half the teachers had delivered only two of the six sessions, that the teacher training day had been cut to a lunch-hour briefing, and that the referral pathway was never set up.

Step 1: Read the outcome result on its own. Knowledge and attitudes barely changed between baseline and follow-up. Taken alone, this looks like a curriculum that failed.

Step 2: Check fidelity. The process evaluation shows the program was not implemented according to its design. The teacher training was cut to a lunch-hour briefing, and one component, the referral pathway, was never set up. A replication without fidelity is not an additional test of the model.

Step 3: Check dosage. Half the teachers delivered only two of the six sessions, so many students received a third of the planned dose. A low dose would be expected to produce little change whatever the curriculum's merits.

Step 4: Draw the conclusion the evidence supports. The result says little about the curriculum and nothing about the theory behind it; it says a great deal about implementation. The superintendent's conclusion is not supported. The sensible next step is to deliver the curriculum with fidelity and a full dose, and evaluate it again.

What this shows. This is the stages-of-change example from earlier in this subsection in different clothes. Process data protect the interpretation of outcome data, and the common mistake is to blame the theory for a failure of delivery.

Process evaluation tells you whether the program happened as designed. Whether it changed anything is the business of outcome and impact evaluation, which the next subsection takes together because they share a method.

3.3 Outcome and Impact Evaluation: Measuring Change Over Time

3.3.1 Outcome evaluation: short-term effects

With outcome evaluation you start looking at the effects of a program. The question is what short-term or immediate effect the program had, and how you answer it depends on your program goals. What did you expect to happen in the short term, and was that expectation guided by theory and by your assessment, for example through the PRECEDE-PROCEED process? The expected outcome has everything to do with theory, assessment, and how these were incorporated into the program design. The method is to collect data on the things you expect to change before the project begins, and then to collect the same information at one or, preferably, more points after the intervention is in place. This is called pre- and posttest data, or baseline and follow-up data. By this we mean that the baseline is the starting point against which change is judged: without a measurement taken before the program, a measurement taken afterward cannot show that anything changed, only where people ended up. The outcome data might be pre- and posttest scores on knowledge, records of health service use, evidence of a policy change, or other educational, ecological, or policy data, and they describe the kind of change a program funded for three or four years can reasonably expect to show. Figure 11.2 shows the timeline that outcome and impact evaluation share, and where the two part company.

Baseline pretest data Intervention process evaluation: records Follow-up 1 Follow-up 2 Outcome evaluation knowledge, attitudes, skills, service use, policy (a three- or four-year program) Years later Impact evaluation behaviour change sustained, morbidity and mortality

Figure 11.2. A timeline shared by outcome and impact evaluation: baseline (pretest) data are collected before the intervention, process evaluation keeps records during it, outcome evaluation measures knowledge, attitudes, skills, service use, and policy at follow-ups 1 and 2 within a three- or four-year program, and impact evaluation measures sustained behaviour change, morbidity, and mortality years later.

In Figure 11.2 the two follow-ups that come soon after the intervention belong to outcome evaluation, while the measurement taken years later belongs to impact evaluation. The method is the same; what changes is what is measured and how long you wait.

3.3.2 Impact evaluation: the health problem itself

Impact evaluation reaches the real health impact of a program, and with it the theoretical links you may have drawn between short-term outcome and long-term impact. By this we mean that impact evaluation asks whether the health problem that justified the program in the first place has actually changed, and so whether the whole chain from early change to health improvement held. For example, consider a smoking intervention with several activities designed to raise awareness of risks and of prevention options. The hypothesis, drawn from the Health Belief Model, is that if a group of people come to see their own susceptibility to cancer from smoking, and see quitting options that do not present an insurmountable barrier, they are likely to quit. The program is therefore linking awareness change to behaviour change, and behaviour change to a reduction in cancer. In this chain, awareness is the short-term outcome, because it can change while the program is running and can be measured at a follow-up soon afterward. Cancer, in contrast, is the long-term impact, because any fall in cancer would appear only years later. Measuring the impact requires a program that stays in place for an extended period (more than three or four years), or the ability to follow participants as a cohort for years, and it means collecting follow-up data on cancer morbidity and mortality rather than on awareness.

3.3.3 The three types compared

Table 11.2 brings the three basic types together. For each type it gives the question, the typical data, the time frame, and what theory and the logic model (introduced in Section 3) say the evaluation checks.

Table 11.2. Process, outcome, and impact evaluation compared by question, typical data, time frame, and what theory and the logic model say.

TypeThe questionTypical dataTime frameWhat theory and the logic model say
ProcessWere the components of the program implemented as planned?Counts of materials, sessions, partners, participants; fidelity and dosage recordsDuring implementationWhether the planned outputs were produced from the inputs
OutcomeWhat short-term or immediate effect did the program have?Pre- and posttest knowledge, attitudes, skills; service utilization; policy changeWithin a program of three or four yearsWhether the predisposing, enabling, and reinforcing factors the theory targeted moved
ImpactHow did the program affect the health problem that was the ultimate target?Sustained behaviour change; morbidity and mortality; health statusMore than three or four years, or a followed cohortWhether the chain from factors to risk to health problem held

Table 11.2 can be used as a decision rule. If an evaluation question counts what was delivered, it is process; if it measures change in knowledge, attitudes, skills, service use, policy, or an early behaviour against a baseline, it is outcome; if it asks about disease, death, or health status over years, it is impact. Interactive 11.2 lets you test that rule on ten evaluation questions drawn from Canadian programs. Read each question, choose process, outcome, or impact, and read the feedback before moving on to the next one.

📊 Interactive 11.2: Which type of evaluation asks this?

Read the question, choose a type, then move to the next one. Ten questions.
Were all eight community kitchen sessions delivered in the Saskatoon neighbourhood, and how many people attended each one?
Six months after enrolling in the provincial cessation program, what proportion of participants report that they have quit smoking?
Fifteen years on, has lung cancer incidence fallen among the cohort of people who used the cessation program?
Did the youth workers complete the stage-of-change assessment with each participant, as the program design required?
Did knowledge of the signs of stroke increase between the baseline survey and the follow-up survey in the campaign region?
Has the rate of new type 2 diabetes diagnoses in the county changed over the ten years since the program began?
How many pharmacies dispensed nicotine replacement products under the program, and how many residents picked them up?
Did the municipal council adopt the smoke-free parks bylaw that the coalition was working toward?
Has the proportion of grade 9 students who say they would seek help for depression increased since the mental health literacy curriculum began?
Did emergency department visits for self-harm among youth in the district fall over the following decade?
Question 1 of 10
Choose the type of evaluation that asks this question.
0 answered

The items in Interactive 11.2 that most often catch people out are the bylaw, which is an outcome because policy change is among the short-term effects a program can produce, and the six-month quit rate, which is an outcome even though quitting is the behaviour that an impact evaluation would follow up years later. Worked Example 11.3 applies all three types to a single Canadian program and asks a question every evaluator meets: what can honestly be reported early? Watch how the time frame of each type decides the answer.

Worked Example 11.3: Evaluating British Columbia's Smoking Cessation Program

The case. Since 2011 British Columbia's Smoking Cessation Program has supplied nicotine replacement products such as patches and gum at no cost through community pharmacies, and has covered prescription cessation medications through the provincial PharmaCare plan. Imagine you have been asked to evaluate it, and the ministry wants "results" within the program's first eighteen months.

Step 1: Identify the program's logic. The program rests on a familiar logic: a barrier (cost) is removed, a cue to action arrives when a pharmacist offers the product, and the province's QuitNow service provides support for the attempt. Barriers and cues to action are Health Belief Model ideas, so the theory predicts more quit attempts and more successful quitting.

Step 2: Write the process questions. A process evaluation would count participating pharmacies, products dispensed, and referrals to counselling, and would check whether pharmacists delivered the brief advice the program design calls for. These records accumulate from the first month.

Step 3: Write the outcome questions. An outcome evaluation would measure quit attempts and self-reported abstinence among participants at follow-up, compared with their smoking at enrolment (the baseline). A six-month follow-up of people who enrolled early can report within the first eighteen months.

Step 4: Write the impact questions. An impact evaluation would ask whether smoking prevalence in the province, and eventually tobacco-related disease, changed over the years that followed. That question needs population data over a long period and cannot be answered by the program's own records.

Step 5: Answer the ministry. Process evaluation can deliver results within eighteen months, and outcome evaluation can deliver early results on quit attempts and abstinence. Impact evaluation cannot, because the long-term impact on tobacco-related disease emerges only over years. The honest explanation uses the distinction between short-term outcomes, the changes the program's logic says come first, and long-term impacts on the health problem itself, which are still to come.

What this shows. The three types are not competing options: one program needs all three, and only the first two can report within a program's early life. The common mistake is to promise impact results on an outcome timetable.

The cessation example used this course's labels, outcome for the short term and impact for the long term. Not every source uses the two words that way, which is the subject of the last subsection.

3.4 Which Word Means Long Term?

The terms outcome evaluation and impact evaluation are attached to different time frames by different sources in the field, even though the underlying idea of short-term and long-term evaluation is the same everywhere. By this we mean that everyone agrees there are early changes and later changes to measure; what differs is which word each source uses for which. You will meet both conventions. In the convention this course uses, which follows much current usage, outcome is short term and impact is long term. However, in Green and Kreuter's PRECEDE-PROCEED framework the labels run the other way: impact evaluation covers the immediate effects on predisposing, enabling, and reinforcing factors and on behaviour, while outcome evaluation covers health and quality of life. This is a source of real confusion, because PRECEDE-PROCEED is also the framework this course uses to plan programs. Some public health evaluation frameworks avoid the problem altogether by speaking of short-, intermediate-, and long-term outcomes. The practical consequence is that the label alone never tells you which kind of change a report measured; for example, a "positive impact" in one report may be the same kind of attitude change that another report calls an outcome.

Exam trap: check the definitions first

In this course, outcome means short term and impact means long term. In PRECEDE-PROCEED the two labels are swapped. A report that calls a change in attitudes an "impact" is not wrong; it is using the other convention. Whenever you read an evaluation report, check its definitions before you compare its results with anyone else's.

This section has distinguished process, outcome, and impact evaluation by the question each answers, the data each needs, and the time frame each requires. The next section puts all three types into a single structure, the logic model, and turns to the methods and pitfalls of making a case that a program caused a change. First, the key takeaways draw this section together, and the knowledge check gives you practice classifying evaluation questions.

Key Takeaways

  • 3.1 Each type of evaluation answers one question, and naming that question is the fastest way to classify a case. Process evaluation asks whether the program's components were implemented as planned, outcome evaluation asks what short-term or immediate effect the program had, and impact evaluation asks how the program affected the health problem that was its ultimate target. A single project may use one, two, or all three. Formative evaluation (used while a program is being developed), cost-effectiveness evaluation, and quality assurance evaluation also exist, but this lesson only names them. In an exam, look at what the evaluation question asks about: whether something "was delivered" points to process, whether something "changed since baseline" points to outcome, and whether a disease "fell over the decade" points to impact.
  • 3.2 Process evaluation keeps records of materials, sessions, partners, and participants and compares what was done with what was planned. Its real value is interpretive. Fidelity data show whether every component was delivered as designed, and dosage data show how much of the program each person received, so together they let you separate a failed theory from a failed delivery, as the stages-of-change example and the Alberta replication in Worked Example 11.2 show. The trap to avoid is concluding that a curriculum or a theory does not work when the process data show it was never fully delivered; a replication without fidelity is not a test of the model.
  • 3.3 Outcome and impact evaluation share one method: collect baseline (pretest) data on what you expect to change before the program starts, then collect the same data at one or more follow-up points. They differ in what they measure and how long they wait. Outcome evaluation measures the short-term changes that theory and assessment predicted, such as knowledge, attitudes, skills, service use, or policy, within a program of three or four years. Impact evaluation measures the health problem itself, such as cancer morbidity and mortality, and needs a program in place for more than three or four years or a cohort followed for years. When a case asks what can be reported early, process and outcome results can be; impact results cannot.
  • 3.4 The labels outcome and impact are not used the same way everywhere. This course, following much current usage, uses outcome for the short term and impact for the long term, while Green and Kreuter's PRECEDE-PROCEED uses impact for immediate effects on factors and behaviour and outcome for health and quality of life. Some frameworks avoid the clash by speaking of short-, intermediate-, and long-term outcomes. On an exam, answer in this course's convention unless a question names another, and if two reports seem to disagree, check their definitions before deciding that either one is wrong.
Knowledge Check: this section

1. A coordinator compares the number of cooking classes actually delivered in each community with the number promised in the proposal, and records how many sessions each participant attended. Which type of evaluation is this, and what related concept do the attendance records support?

Outcome evaluation measures short-term change in participants from baseline to follow-up. Counting classes delivered against classes promised measures what the program did, not how participants changed.
Comparing classes delivered with classes promised is the basic process question, answered by record keeping. Attendance per participant gives dosage, the amount of the intervention each person received.
Impact evaluation follows the health problem itself over years, often through a cohort. Class counts and attendance concern delivery during implementation, not long-term change in health.
Formative evaluation, such as pretesting, shapes a program during development, before or as it rolls out. This coordinator is checking delivery of a program already running against its proposal.
Correct answer: process evaluation; dosage. The information needed is the description of process evaluation and the dosage card in 3.2. Comparing the classes delivered with the classes promised is the basic process question, answered by record keeping, and recording each participant's attendance gives dosage, the amount of the intervention each person received. Outcome evaluation with baseline data is the tempting distractor, but nothing here measures change in participants; it measures what the program delivered. (Section 2, 3.2 Process Evaluation: Did We Do What We Planned?; Objective 3.2)

2. A stage-based diet program shows weaker results than expected. Process data reveal that staff skipped the periodic stage assessments that the program design required. What does this tell the evaluator?

A program not delivered as designed is not a test of its theory. Because staff skipped the stage assessments, the result says little about the Transtheoretical Model.
The program's components were tied to each participant's stage, so skipping the stage assessments means it was not implemented as designed. Process data point to delivery rather than theory or design as the likely problem.
Nothing in the case concerns the baseline. The process data identify a specific failure of implementation, which is the explanation the evaluator should consider first.
Process, outcome, and impact are types of evaluation, not categories of program. The finding is about implementation fidelity, which process evaluation exists to detect.
Correct answer: the problem may lie in implementation rather than in the theory. The information needed is the stages-of-change example in 3.2. The program's components were tied to each participant's stage, so skipping the stage assessments means the program was not delivered as designed, and the process data point to implementation. Saying the Transtheoretical Model has been disproven is the trap: a program that was not implemented with fidelity is not a test of its theory. (Section 2, 3.2 Process Evaluation: Did We Do What We Planned?; Objective 3.2)

3. A provincial cessation program has been running for two years. Which of the following is an impact evaluation question in the sense this course uses?

Counting pharmacies that dispensed products measures what the program delivered, which is process evaluation, not a change in the health problem.
A six-month quit rate is a short-term change measured from baseline, which this course calls an outcome, even though quitting is the behaviour an impact evaluation would follow for years.
Perceived susceptibility is a Health Belief Model factor that changes early, so a change from baseline is a short-term outcome, not an impact.
Impact evaluation asks how the program affected the health problem that was its ultimate target, over an extended period. Tobacco-related disease across fifteen years is exactly that.
Correct answer: whether tobacco-related disease declined over fifteen years. The information needed is the distinction in 3.3 and Table 11.2. Impact evaluation asks how the program affected the health problem that was its ultimate target, measured over an extended period. The six-month quit rate is tempting because quitting is the behaviour the program wants, but a short-term change measured from baseline is an outcome; pharmacy counts are process data, and belief change is also an outcome. (Section 2, 3.3 Outcome and Impact Evaluation: Measuring Change Over Time, Table 11.2; Objective 3.3)

4. Two reports evaluate similar school programs. One, following the convention used in this course, calls the change in students' attitudes an "outcome"; the other, following PRECEDE-PROCEED, calls the same change an "impact." What does the difference in terminology imply for a reader?

The second report is not wrong. PRECEDE-PROCEED uses impact for immediate effects on factors and behaviour, so it is following the other convention, not making an error.
Impact does not always mean health status. In PRECEDE-PROCEED impact covers immediate effects on factors and behaviour, so both reports may have measured the same kind of attitude change.
This course uses outcome for the short term and impact for the long term, while PRECEDE-PROCEED swaps them. A reader should check each report's definitions before comparing results.
Registries rate the quality of evidence, not the label a report attaches to its results. The word chosen reflects a convention, not the quality of the study.
Correct answer: check each report's definitions before comparing. The information needed is the discussion of labels in 3.4. This course uses outcome for the short term and impact for the long term, while PRECEDE-PROCEED attaches the labels the other way around, so the two reports may be describing the same kind of change. Discounting the second report is the tempting error; it is not wrong, it is using the other convention. (Section 2, 3.4 Which Word Means Long Term?; Objective 3.4)

5. An Ontario public health unit launches a three-year program in which public health nurses visit new parents at home to promote safe infant sleep, using modelling and skill building from Social Cognitive Theory. Which of these is a process evaluation question for the program?

Process evaluation compares what was done with what was planned. Checking that visits were made and that modelling and practice components were delivered as designed is a question about implementation and fidelity.
Parents' confidence is self-efficacy, a factor the program targets, measured from first visit to follow-up. Short-term change from a baseline makes this an outcome question.
Reported sleep practices compared with baseline measure short-term change in behaviour among participants, which this course calls an outcome, not delivery of the program.
Sleep-related infant deaths are the health problem itself, measured against prior years, which makes this an impact question.
Correct answer: whether the planned visits were made and the components delivered as designed. The information needed is the process question in 3.1 and the method in 3.2: process evaluation compares what was done with what was planned, using records such as visit logs and checklists of components delivered. The confidence and practice questions are tempting because they concern the program's participants, but they measure short-term change from baseline, which makes them outcomes; the infant-death question is an impact. (Section 2, 3.1 Three Questions and 3.2 Process Evaluation: Did We Do What We Planned?; Objective 3.1)

6. For the same home-visiting program, which evaluation question could NOT be answered before the three-year funding ends, and why?

Visit logs are process records kept throughout implementation, so whether nurses completed the planned visits can be answered from the first months.
Self-efficacy can be measured, and changes in it are a short-term outcome that baseline and follow-up surveys can capture within three years.
Infant deaths are the health problem itself, so this is impact evaluation, which needs a program in place longer than three or four years or a followed cohort. Rare events lengthen the wait further.
Outcome evaluation relies on follow-up surveys compared with baseline. Changes in reported sleep practices are exactly the short-term outcome such surveys can capture within three years.
Correct answer: whether sleep-related infant deaths fell. The information needed is the time frames in 3.3. Infant deaths are the health problem itself, so this is an impact question, and impact evaluation needs a program in place for an extended period or a cohort followed for years; with rare events the wait is longer still. Self-efficacy and reported practices are outcomes that baseline and follow-up surveys can capture within three years, and visit logs are collected throughout the program. (Section 2, 3.3 Outcome and Impact Evaluation: Measuring Change Over Time; Objective 3.3)

✦ Pass the knowledge check with 100% to continue

Section 3

Logic Models, Methods, and Confounds

⏱ Estimated reading time: 18 minutes

Section 3 of 4

Logic Models, Methods, and Confounds

Setting up an evaluation, and making a case that will hold.

The logical chain

Back up the chain

Factors

Predisposing, enabling, reinforcing: what the intervention targets.

Risks

Behavioural and environmental risk factors.

Health problem

Morbidity and mortality in the affected group.

Context

Social conditions; administrative and policy resources.

The parts

Five linked components

Inputs: resources and staffProblem / goal: the health problemOutputs: activitiesShort-term outcomesLong-term impactsIndicators and measures

Each column is also a type of evaluation: process for outputs, outcome for the short term, impact for the long term.

Worked example

Harfield County

Outputs

Outreach; diet classes and cooking groups through churches and veterans centres; diet modelling by leaders.

Short-term outcomes

Knowledge of food risk and improved diet; skills to choose healthier diets.

Long-term impact

Reduced incidence of diabetes and its consequences.

Methods

A ladder of designs

  • Historical record keeping: are inputs producing the planned outputs?
  • Qualitative approaches: what changed, from whose viewpoint, and why?
  • Periodic inventory / time series and benchmarking: is progress being made, and how does it compare?
  • Quasi-experimental: a similar community that did not get the program.
  • Classic experimental: random assignment to intervention or control.
Confounds

Seven ways a case gets weaker

HistoryMaturationTestingRegression to the meanSelection biasMortality / attritionDiffusion of treatment

The intervention is the independent variable; dependent variables are what the program tries to change (diet, exercise).

Fit

No cookie-cutter standard

Quantitative

Numbers collected the same way from many people: comparison and generalization.

Qualitative

Interviews, focus groups, observation: meaning, context, and the dynamics of change.

Process when getting started is the achievement; outcome when time is short; impact when the program is long enough to show it.

Carry forward

What to take into the next section

  • A logic model gives each type of evaluation a column, and indicators make the columns measurable.
  • Designs trade rigour against cost, feasibility, and ethics; confounds are the alternatives a design must rule out.
  • Fit the evaluation to the program; qualitative and quantitative data answer different questions.
  • Next: the career settings where theory and evaluation are put to work, in general and in Canada.

Overview

The last section separated three types of evaluation. This one puts them into a single structure, the logic model, which grows out of the PRECEDE-PROCEED planning framework, with its sequence of assessments leading to a program and its evaluation. It then turns to the hardest question in the lesson: how do you make the case that your program, and nothing else, caused the change you observed? The answer is a ladder of designs, a list of confounds that every design must anticipate, and a reminder that the right evaluation is the one that fits the program.

Learning Objectives

  • 4.1 Explain how the PRECEDE assessments form a logical chain that an intervention, and its evaluation, travel back up.
  • 4.2 Place program elements into the components of a logic model (inputs, problem, outputs, short-term outcomes, long-term impacts, indicators and measures) and name the type of evaluation that tests each column.
  • 5.1 Compare evaluation designs from historical record keeping to the classic experiment, stating what each can claim and the problems each brings, and distinguish empowerment evaluation from them.
  • 5.2 Identify the seven confounds that weaken a causal claim, and distinguish independent from dependent variables.
  • 5.3 Explain why no single evaluation design suits every program, and what quantitative and qualitative data each contribute.

4.1 From PRECEDE to a Logical Chain

Recall the PRECEDE assessments: social (the community context and quality-of-life issues), epidemiological (the health problems and affected group, from morbidity and mortality data), behavioural and environmental (the risk factors behind those problems), educational and ecological (norms, attitudes, awareness, and policies understood as predisposing, enabling, and reinforcing factors), and administrative and policy (resources, community politics, and structures that help or hinder implementation). Can you see the logical chain? Health problems result from risks, which result from predisposing, enabling, and reinforcing factors, within a community and policy context. By this we mean that the assessments, read in order, describe a sequence of causes: each step explains the one before it. The assessment works down the chain, starting from the health problem and asking what lies behind it. Real situations rarely line up so neatly, but for designing an intervention and its evaluation the chain is useful, because the intervention goes back up it in the opposite direction. You target several factors, which should affect risks, which should affect the health problem as recorded in epidemiological data. For example, a program might raise knowledge about diet (a predisposing factor), which should improve diet (the risk behaviour), which should in time reduce a diet-related disease (the health problem). The order matters for evaluation, because the factors change first and the health problem last. Figure 11.3 shows this PROCEED side of the logic.

Intervention Educational / ecological change predisposing, enabling, reinforcing Behavioural / environmental change risk factors Change in epidemiology Change in social conditions Administrative / policy change and resources people, funds, community politics, structures that help or hinder implementation Process evaluation covers the intervention box; outcome evaluation the educational / ecological box; impact evaluation the rest.

Figure 11.3. Intervention linkages on the PROCEED side of the logical chain: the intervention leads to educational and ecological change in predisposing, enabling, and reinforcing factors, then to behavioural and environmental change in risk factors, change in epidemiology, and change in social conditions, supported by administrative and policy change and resources; the note at the top of the diagram assigns the intervention to process evaluation, the educational and ecological change to outcome evaluation, and the rest to impact evaluation.

Read from left to right, Figure 11.3 is a map of evaluation as well as of change. The note at the top assigns the intervention to process evaluation, the change in predisposing, enabling, and reinforcing factors to outcome evaluation, and the later changes in risk factors, epidemiology, and social conditions to impact evaluation, so every link in the chain is something an evaluation can check. A logic model turns this chain into a planning structure, which is the subject of the next subsection.

4.2 The Parts of a Logic Model

A logic model is a diagram or structure that links what you plan to do with its expected outcomes and impacts. By this we mean that it sets out, on one page, the argument that connects the program's resources and activities to the changes it hopes to produce. The purpose of drawing it is to make every link in that argument visible, so that planners can check that it makes sense and evaluators can see what to measure. A logic model has five linked components.

The first is the health problem, which comes from the epidemiological assessment (HIV, cancer, malaria, diabetes, and who it affects) and may name the contributing factors you intend to address. The second is the set of outputs: the activities you plan, such as materials, events, screenings, or trainings, and the factors each is meant to change. The choice of outputs is guided by the factors identified and the theory you believe appropriate, so outputs are where the theory enters the model. The third is the set of inputs, the resources, staff, components, and funds invested, some of them identified in the administrative and policy assessment. Inputs and outputs are easily confused. The simplest way to keep them apart is that inputs are what the program has or uses, while outputs are what the program does or delivers.

The fourth component is the projected outcomes and impacts. The outcomes are the short-term effects hypothesized for the outputs, typically enabling and predisposing factors such as knowledge, while the impacts are the longer-term effects on risk behaviour and health status. The word "hypothesized" matters here. By this we mean that the outcomes column is a prediction drawn from the theory: it states what the planners expect the activities to change if the theory is right. For example, if a diet class is meant to raise knowledge of healthy food choices, then higher knowledge is the hypothesized outcome, and less diet-related disease years later is the projected impact. The fifth component is the indicators and measures, the criteria and tools for measuring each of these: health data, survey or qualitative data, measures of skill. Indicators turn each prediction into something that can be observed. Without them, an outcome such as "better diet knowledge" remains a hope rather than something that can be checked, whereas a knowledge survey given at baseline and follow-up shows whether the hope came true.

The logic model also connects directly to the three types of evaluation from Section 2, because each column is tested by one of them. Process evaluation tests the outputs column, by asking whether the planned activities were delivered. Outcome evaluation tests the short-term column, and impact evaluation tests the long-term column. The indicators are what make each column measurable. In this way the logic model turns the abstract distinction between process, outcome, and impact into a practical plan for what to measure at each stage.

Worked Example 11.4 builds a logic model step by step for Harfield County, a fictional rural county, starting from its assessment. Watch how each column follows from the assessment and the theory chosen, rather than from a list of activities the planners happened to like.

Worked Example 11.4: A Logic Model for Harfield County

The case. Harfield County is a fictional rural county where type 2 diabetes is rising among adults and young adults. Large supermarkets are scarce, the newspaper rarely covers health, and the county's strength is its churches, service organizations, and a veterans group, full of people who are leaders or at least "movers."

Step 1: Read the assessment. The problem is type 2 diabetes linked to diet. Two possible channels are weak: healthy food is hard to reach, and the local newspaper will not carry health messages. One asset is strong: organized groups with trusted leaders.

Step 2: Choose the theory. Because the county's strength is its leaders and organizations, the planners choose a social-cognitive approach with social network components. Leaders can model healthier diets, which is Social Cognitive Theory's observational learning, and the organizations people already belong to can carry the classes and cooking groups, which is a social network strategy.

Step 3: Fill in the columns. Table 11.3 sets out the resulting logic model in five columns, from inputs to long-term impacts.

Table 11.3. Logic model for the fictional Harfield County program on type 2 diabetes, built on a social-cognitive approach with social network components.

InputsProblem / goalOutputsShort-term outcomesLong-term impacts
Church and veterans organization staffType 2 diabetes caused by dietCommunity outreach; diet classes and cooking groups through churches and veterans centres; diet modelling by community leadersKnowledge and awareness of food risk for diabetes; knowledge of improved diet practices; skills to choose healthier dietsReduced incidence of diabetes and its consequences

Step 4: Read the model against the theory and the evaluation. Modelling by leaders is observational learning; delivery through churches and veterans centres is the social network strategy; knowledge and skills are the outcomes the theory predicts; diabetes incidence is the impact. Each column then names its evaluation and its indicators: process evaluation counts the classes and cooking groups held and who attended, outcome evaluation compares diet knowledge and skills at baseline and follow-up, and impact evaluation tracks diabetes incidence in health data over years.

What this shows. A logic model is the assessment and the theory written out as a chain that an evaluator can test column by column. The common mistake is to list activities first and add a theory afterward, which leaves the outcomes column with nothing specific to measure.

Now build one yourself. Interactive 11.3 applies the same five columns to a Canadian school district's mental health literacy curriculum. Click a component, then click its column; when all fifteen components are placed, the completed model shows the indicators and measures, and the type of evaluation that tests each column.

📊 Interactive 11.3: Build the logic model.

Click a component, then click its column. When all fifteen are placed, the model reveals its indicators and the type of evaluation that tests each column.
The case. A Canadian school district finds rising anxiety and depression symptoms among grade 9 students and very low help-seeking. Guided by Social Cognitive Theory and the school as a setting, it adopts a mental health literacy curriculum delivered by trained teachers, in partnership with the local Canadian Mental Health Association branch.
Inputs
resources and staff
Problem / goal
the health problem
Outputs
activities
Short-term outcomes
of the activities
Long-term impacts
health status
Select a component to begin.
0 of 15 placed
Indicators and measures, and the evaluation that tests each column
ColumnIndicator or measureEvaluation
OutputsSessions delivered per class; number of teachers trained at the workshop; parents attending; referral pathway in useProcess (fidelity, dosage)
Short-term outcomesKnowledge, stigma, and help-seeking intention scales at baseline and after the curriculum; counsellor referralsOutcome
Long-term impactsEmergency department records for self-harm; district survey data on untreated depression; graduation records, over yearsImpact

The trickiest placements in Interactive 11.3 are the teacher training workshop, which is an output even though the trained teachers it produces become an input to the classroom sessions, and low help-seeking, which belongs in the problem column as a contributing factor the program intends to address. Once a logic model says what should change, the evaluator faces the next question: how strong a case can be made that the program caused the change?

5.1 Evaluation Designs: Making a Case

This lesson does not present methods in detail, but their shared rationale is worth stating. Evaluation methods are tools for making a case for the effectiveness, or lack of effectiveness, of an intervention; it is a little like being a lawyer. By this we mean that an evaluator, like a lawyer, rarely has absolute proof and instead assembles evidence that makes one explanation more convincing than the alternatives. Each method supplies information that could support the claim that the intervention was responsible for a change, and the more rigorous the method, the more you can argue that the intervention, and nothing else, was responsible. As designs get more rigorous they add control groups (people who do not get the intervention) and, at the top, random assignment. The purpose of a control group is to show what would probably have happened without the program, so that the difference between the two groups can be credited to the intervention. The first six climb from the historical recordkeeping approach to the classic experimental design, giving for each what you do, the data, the question it answers, and its problems; the seventh describes empowerment evaluation, which serves a different purpose.

Historical recordkeeping approach (process)

What you do: track project activities and client participation. Data: sessions conducted, services delivered, calls taken, advertisements posted, client utilization. Question: are program inputs creating the planned outputs? This is the floor beneath every other design.

Qualitative evaluation approaches (any type)

What you do: assess what participants and staff have experienced in terms of change. Data: interviews, focus groups, and observations using semistructured guides. Questions: what is the nature of the change that has or has not occurred, from the viewpoint of implementers and participants? Was it the intended change? What factors were involved? Especially useful when randomized designs are not possible and risk behaviour must be understood in context.

Periodic inventory / time series (process and outcome)

What you do: at set intervals, beginning with a baseline, tabulate program data and complete surveys to check progress toward goals, such as the number who quit smoking or changed their attitude toward quitting. There is no comparison group, which means the design can show that participants changed over time but cannot show what would have happened to them without the program. Questions: are short-term goals being attained? Is progress being made?

Benchmarking (process and outcome)

What you do: assess program and participant data against a comparable benchmark, such as national or population-specific data. A Canadian program might compare its quit rate with Canadian Tobacco and Nicotine Survey figures, or its screening uptake with the provincial average. Question: are program results comparable to results documented elsewhere?

Quasi-experimental design (outcome and impact)

What you do: identify a similar community or population sample that does not receive the intervention, and collect pre- and posttest or time series data from both. The comparison community stands in for what would have happened without the program, so change there can be set against change in the program community. Problems: are the two groups really comparable? Did the comparison group get any of the intervention (contamination)? Did anything else happen in the intervention community? Question: are changes the result of the intervention, or would they have occurred anyway?

Classic experimental design (outcome and impact)

What you do: randomly assign individuals to the intervention or to serve as controls, for example every other name on a list, then collect pre- and posttest or time series data from both. Problems: did controls receive some of the intervention by accident? Is it ethical to provide an intervention only to some people in a community that needs it? A program that includes a full-scale experimental design can be viewed as an evaluation project.

Empowerment evaluation (a different purpose)

Drawing on the community theories of Paulo Freire and other participatory approaches, empowerment evaluation is not an experimental design meant to produce rigorous data. It is a collaborative effort with the community to identify goals and decide how to measure them, often instead of standard validated instruments, so that the community improves its program on its own terms rather than by outside models. In Canada it sits naturally with the First Nations principles of ownership, control, access, and possession (OCAP) for community information, and with Indigenous-led evaluation.

Read as a ladder, the designs trade feasibility for rigour. By this we mean that each step up allows a stronger claim that the program caused a change, but each step also costs more and is harder to carry out in a real community. At the bottom, historical record keeping asks only whether inputs are producing the planned outputs, so it can support a claim about delivery but not about effect. Qualitative approaches ask what changed, from whose viewpoint, and why, which gives depth but no comparison. Periodic inventories add a baseline and repeated measurement, so they can show that participants changed over time. Benchmarking adds an outside comparison, so the program's results can be set beside results documented elsewhere. However, none of these designs can rule out the possibility that the change would have happened anyway.

The next two steps address that possibility directly. A quasi-experimental design adds a similar community that does not receive the program, so that change in the program community can be compared with change in a community without it. This raises its own problems: whether the two communities are really comparable, whether the comparison group received some of the intervention (contamination), and whether other events happened in the intervention community. The classic experiment goes one step further and adds random assignment of individuals, which answers the worry about comparability because chance, rather than the evaluator or the participants, decides who receives the program. In turn it raises the problems of accidental exposure of controls and the ethics of providing an intervention to only some people in a community that needs it; a program that includes a full-scale experimental design can be viewed as an evaluation project.

Empowerment evaluation, in contrast, steps off the ladder altogether, because it is not trying to make the strongest case for cause and effect. Drawing on Paulo Freire's community theories and participatory programming, it is a collaborative effort in which the community identifies its goals and how to measure them, often instead of standard validated instruments, so that the community improves its program on its own terms. The question it answers is therefore different: not "did the program cause this change?" but "is the program achieving what this community values, and how can the community improve it?"

These designs also connect to the history of positivist social science. The methods now regarded as rigorous rest on Enlightenment assumptions about isolating a single cause. By this we mean the idea that, if everything else is held constant, the effect of one factor can be observed on its own, as in a laboratory experiment where only one condition is varied. A community is not a laboratory, however. People talk to each other, news and policies change during the program, and participants live lives the evaluator cannot control. It is therefore fair to ask when a quasi-experimental or classic experimental design truly isolates the effect of a program, and how else you could know whether a program made an impact. Qualitative and empowerment approaches answer that question differently. They trade the ability to isolate a cause for a richer account of what changed and why, drawn from the people who lived through the program. Whatever design you choose, the case can be weakened by alternative explanations, which the next subsection names.

5.2 Confounds: What Weakens the Case

When you argue that a program caused a change, you must also address what could weaken the case. Some weaknesses are methodological, such as a design without a comparison group. Others are confounds: unplanned occurrences or other issues that complicate the claim that only the intervention caused the change. By this we mean that a confound offers a rival explanation for the change you observed. If something other than the program could have produced the same before-and-after difference, then the difference cannot be credited to the program alone. Seven confounds are listed here, and designs often build in ways to offset them. Table 11.4 names each confound, says what it is, and gives a Canadian illustration.

Table 11.4. Seven confounds that complicate the claim that only the intervention caused a change, with a Canadian illustration of each.

ConfoundWhat it isA Canadian illustration
HistoryAn unconnected event during the program causes change; the standard example is a famous film star dying of AIDS.A federal tobacco tax increase arrives midway through a cessation program.
MaturationParticipants change simply because they gain life experience during the program.Grade 9 students in a two-year program become grade 11 students; some change in risk-taking is growing up.
TestingChange from baseline to follow-up reflects familiarity with the survey or its questions.Parents answer a safe-sleep questionnaire at three home visits and learn the "right" answers.
Regression to the meanExtreme starting values move toward the average regardless of the program; with extreme poverty or very high-risk behaviour, scores may improve only because they cannot get worse.A campus program recruits the heaviest drinkers; their drinking falls whether or not the program worked.
Selection biasParticipants are not representative, for example because answering is voluntary, so responses come only from the motivated.An online follow-up survey is completed mainly by participants who liked the program.
Mortality or attritionParticipants leave over time, so follow-up data come only from those who stayed.Half of a cessation cohort is lost to follow-up; those who relapsed are least likely to answer.
Diffusion of treatmentThe control group receives some or all of the intervention, for example when radio advertisements reach the control community, so the comparison shows little difference.A campaign in one Lower Mainland municipality is seen by residents of the neighbouring comparison municipality who commute through it.

Each confound in Table 11.4 offers a rival story. History and maturation explain a change by what happened to participants during the program, testing by what happened to them because of the survey, regression to the mean by where they started, selection bias and attrition by who answered, and diffusion of treatment by what happened to the comparison group. Regression to the mean is the least intuitive of these, so it is worth spelling out. People are often recruited into a program because their scores are extreme at one moment, and extreme scores partly reflect a bad week or a bad month that will not last. When they are measured again, many will have drifted back toward their usual level whether or not the program did anything. For example, the heaviest drinkers on a campus would, on average, report drinking somewhat less a few months later even with no program at all, so a before-and-after drop in their drinking cannot be credited to the program alone. Diffusion of treatment works in the opposite direction. Instead of making a program look better than it is, it makes the program look weaker, because when the comparison group also receives some of the intervention, the difference between the two groups shrinks. Confounds are easier to reason about once two terms are clear, because a confound is anything other than the intervention that could move the thing the program is trying to change.

Definitions to know: independent and dependent variables

An independent variable is a variable presumed to influence the outcome. In an evaluation the main independent variable is the intervention itself: whether, and how much of, the program a person received. Characteristics of a person or situation that the intervention is not trying to change, such as being a woman or a recent immigrant, are usually treated as covariates or used to compare subgroups, and they tell a great deal about who and what the program can change. A dependent variable is what the intervention is trying to change, and what outcome or impact evaluation measures. For example, consider an intervention to change diet and exercise among women who are recent immigrants. The independent variable is taking part in the intervention, because that is what the evaluation presumes will make a difference. Being a woman and being a recent immigrant are fixed characteristics: the program does not try to change them, so they describe who took part rather than what the program did. The dependent variables are diet and exercise behaviour, because those are what the program is trying to change.

Knowing the threats does not mean every evaluation must neutralize all of them, or that only the most rigorous design is worth doing, as the last subsection explains.

5.3 No Cookie-Cutter Standard

Different evaluations give different results, and results short of a classic experiment are not without merit. By this we mean that the ladder of designs in 5.1 is not a scale on which only the top rung counts. Programs run in the real world, and success has many levels, so the right evaluation is the one that matches the level of success a program can realistically reach. For example, when getting a program in place and operating is itself an achievement, a process evaluation may be all you need. When the time period is short, or barriers are many and only a limited effect can be expected, an outcome evaluation may be the best you can do. When longer-term impacts are realistic, because the program runs for ten years or includes a long-term follow-up, look for them. In the same way, the kind of program shapes the kind of result: a clinical intervention calls for clinical outcomes.

The kind of data matters too, and the two broad kinds answer different questions. Quantitative data can be expressed numerically and analyzed statistically: fixed-choice survey answers, official demographic or epidemiological records. Because they are collected the same way from many people, they support comparison and generalization; they can show, for example, how much a group changed and whether that change is larger than in another group. Qualitative data, in contrast, are narrative, descriptive, and subjective: extended interviews, focus groups, and observation, where people have time to explain their actions, beliefs, and interpretations. They are gathered from fewer people, but they reveal meaning, context, and the dynamics of interactions such as those between patient and doctor. In other words, quantitative data are strongest at saying how much changed, while qualitative data are strongest at explaining what the change meant and why it did or did not happen. The conclusion is that there is no cookie-cutter standard. Evaluation is tailored to the situation, the program, and the evidence needed, and a reader judging an evaluation should ask whether its design fits the program, not only whether it reached the top of the ladder.

Worked Example 11.5 brings the logic model, the ladder of designs, and the confounds together in one realistic case, a one-year social prescribing pilot with no money for a comparison group. Watch how the design is chosen to fit the program, and how the claim is limited to what the design can support.

Worked Example 11.5: Designing the Evaluation of a Social Prescribing Pilot

The case. A network of community health centres in Ontario pilots social prescribing for one year. Primary care providers refer patients who are isolated, lonely, or coping with chronic conditions to a link worker, who connects them with walking groups, arts activities, and volunteering. There is no money for a comparison group, the centres serve very different neighbourhoods, and patients join whenever their provider refers them.

Step 1: Lay out the logic model. The model runs from referrals and link-worker connections (outputs) to reduced loneliness and greater confidence in managing health (short-term outcomes) to fewer primary care and emergency visits (long-term impacts). A one-year pilot can realistically reach the outcomes, not the impacts.

Step 2: Choose a realistic design. With no comparison group, a quasi-experimental design is out of reach, and a classic experiment would mean randomly refusing referrals. A periodic inventory fits: baseline and follow-up surveys for each patient, benchmarked where possible against provincial data, plus interviews with patients and link workers.

Step 3: State what the design can claim. It can claim that participants improved, not that the program alone caused the improvement.

Step 4: Name the confounds. Regression to the mean threatens the claim, since providers refer the most isolated patients, whose scores are likely to move toward the average anyway. Mortality or attrition threatens it too, since those who did not benefit stop attending and drop out of follow-up.

Step 5: Add qualitative data. Interviews would reveal what "connection" meant to participants and why some referrals never became attendance, which surveys cannot show.

What this shows. A modest design, honestly described, is better than a rigorous design that cannot be carried out. The common mistake is to claim that a program caused a before-and-after change in a group selected for its extreme scores.

This section has built the logic model from the logical chain of PRECEDE assessments, compared evaluation designs from record keeping to the classic experiment, named the confounds that a claim of effect must address, and argued that the evaluation should fit the program. The final section asks where, in a career, all of this gets used. The key takeaways and the knowledge check come first.

Key Takeaways

  • 4.1 The PRECEDE assessments, social, epidemiological, behavioural and environmental, educational and ecological, and administrative and policy, line up into a logical chain. Health problems result from risks, which result from predisposing, enabling, and reinforcing factors, within a community and policy context. An intervention travels back up that chain: it targets the factors, which should change the risks, which should change the health problem in the epidemiological data. That is the PROCEED side shown in Figure 11.3, and every link in it is something an evaluation can check. If asked what a program changes first, the answer is the predisposing, enabling, and reinforcing factors, not the disease.
  • 4.2 A logic model links what a program plans to do with its expected results: inputs (resources, staff, partners, funds), the problem (from the epidemiological assessment, with contributing factors), outputs (the activities delivered), short-term outcomes, long-term impacts, and the indicators and measures for each. Each column is also a type of evaluation, with process evaluation for outputs, outcome evaluation for the short-term column, and impact evaluation for the long-term column. In Harfield County (Worked Example 11.4), church and veterans staff are inputs, cooking groups are outputs, diet knowledge and skills are outcomes, and lower diabetes incidence is the impact. The usual trap is confusing inputs with outputs: what the program invests is an input, and what it does or delivers is an output.
  • 5.1 Evaluation methods are tools for making a case, a little like a lawyer's, and the more rigorous the method, the more you can argue that the intervention, and nothing else, caused the change. Designs climb from historical record keeping through qualitative approaches, periodic inventories or time series, and benchmarking, to quasi-experimental designs with a similar comparison community and the classic experiment with random assignment. Each rung adds a way to rule out alternatives and brings its own problems: comparability and contamination for the quasi-experiment, and accidental exposure of controls and the ethics of withholding a program for the experiment. To identify a design in a case, ask whether there is a comparison group and whether it was formed by random assignment. Empowerment evaluation is different in kind, because the community defines its goals and measures in order to improve its program on its own terms.
  • 5.2 Confounds are unplanned occurrences or other issues that complicate the claim that only the intervention caused a change. The seven to know are history (an outside event), maturation (participants gaining life experience), testing (familiarity with the survey), regression to the mean (extreme starting values drifting toward the average), selection bias (unrepresentative participants or respondents), mortality or attrition (follow-up data only from those who stayed), and diffusion of treatment (the control group receiving some of the intervention). The intervention is the main independent variable, fixed characteristics such as being a recent immigrant are usually treated as covariates, and the dependent variables are what the program tries to change. In a case, ask what else could explain the change, and match that rival story to the confound that names it.
  • 5.3 There is no cookie-cutter standard: results short of a classic experiment are not without merit, because programs run in the real world and success has many levels. A process evaluation may be enough when getting a program running is the achievement, an outcome evaluation may be the best you can do when time is short, and impacts are worth seeking when a program runs for ten years or includes long-term follow-up. Quantitative data, collected the same way from many people, support comparison and generalization, while qualitative data, gathered from fewer people in more depth, reveal meaning, context, and the dynamics of interaction. When asked to choose a design for a case, fit it to the program and the evidence needed, and limit the claim to what that design can support, as Worked Example 11.5 does.
Knowledge Check: this section

1. In a logic model for a school mental health program, where does "six classroom sessions delivered by trained teachers" belong, and which type of evaluation tests that column?

Inputs are what the program has or uses, such as funds and trained teachers. The six sessions are what the program does or delivers, which makes them outputs.
Sessions are activities the program delivers, which a logic model calls outputs, and process evaluation tests the outputs column by asking whether the planned activities were produced.
Short-term outcomes are the effects of the activities, such as higher mental health knowledge. The sessions themselves are the activities, not their effects.
Long-term impacts are effects on health status, such as fewer emergency visits for self-harm. Classroom sessions are activities, the outputs column, tested by process evaluation.
Correct answer: outputs; process evaluation. The information needed is the list of logic model components in 4.2. Sessions are activities the program delivers, which a logic model calls outputs, and process evaluation asks whether the planned outputs were produced from the inputs. Inputs is the tempting distractor because trained teachers are mentioned, but the teachers themselves would be an input, a staff resource; the sessions they deliver are outputs, and knowledge change would be an outcome. (Section 3, 4.2 The Parts of a Logic Model and Interactive 11.3; Objective 4.2)

2. An evaluator compares a cessation program's community with a similar neighbouring community that did not receive the program, collecting baseline and follow-up data in both, but without random assignment. Which design is this, and which problem does it raise?

Benchmarking compares program results with outside data such as national figures. This evaluator collects baseline and follow-up data in a similar community that did not get the program.
A classic experiment randomly assigns individuals to intervention or control. Without random assignment, a comparison with a similar community is a quasi-experimental design.
A similar community that does not receive the program, with baseline and follow-up data in both but no random assignment, is quasi-experimental. Its problems are comparability and contamination of the comparison group.
A periodic inventory tracks progress from a baseline with no comparison group. This design adds a comparison community, which moves it up the ladder.
Correct answer: quasi-experimental design, with problems of comparability and contamination. The information needed is the ladder of designs in 5.1. A similar community that does not receive the program, with baseline and follow-up data in both but no random assignment, is the quasi-experimental design, and its characteristic problems are whether the communities are truly comparable and whether the comparison community received some of the intervention. Classic experimental design is tempting, but it requires random assignment of individuals. (Section 3, 5.1 Evaluation Designs: Making a Case; Objective 5.1)

3. During a youth drug-use program, a widely reported overdose death of a well-known musician shifts attitudes among young people in both the intervention and comparison communities. Which confound is this?

History is an unconnected event during the program that causes change, like the film star dying of AIDS in the standard example. A widely reported overdose death shifting attitudes plays the same role.
Maturation is change because participants gain life experience during the program, such as growing older. Here a single outside event, not growing up, shifted attitudes.
Testing is change that reflects familiarity with the survey or its questions. The shift here came from a public event outside the program, not from repeated measurement.
Regression to the mean is extreme starting values drifting toward the average. Nothing suggests participants were chosen for extreme scores; an outside event changed attitudes.
Correct answer: history. The information needed is Table 11.4 in 5.2. History is an event unconnected to the program that occurs during it and produces change; the standard example is a famous film star dying of AIDS, and a widely reported overdose death plays the same role. Maturation is tempting because the participants are young, but it describes change from gaining life experience during the program, not from an outside event. (Section 3, 5.2 Confounds: What Weakens the Case, Table 11.4; Objective 5.2)

4. An intervention aims to change diet and exercise among women who are recent immigrants. Which is a dependent variable?

Being a woman is a fixed characteristic the program does not try to change, usually treated as a covariate. It describes who took part, not what the program aims to change.
Being a recent immigrant defines the target group but is not something the program tries to change, so it is a covariate, not a dependent variable.
The neighbourhood is a characteristic of participants' situation that the intervention does not aim to change. A dependent variable is what the program is trying to change.
A dependent variable is what the intervention is trying to change and what outcome or impact evaluation measures. The program aims to change diet and exercise, so exercise behaviour is a dependent variable.
Correct answer: exercise behaviour. The information needed is the definitions callout in 5.2. A dependent variable is what the intervention is trying to change, and what an outcome or impact evaluation measures, here diet and exercise behaviour. Being a woman or a recent immigrant is tempting because those traits define the target group, but they are characteristics the program does not try to change, usually treated as covariates; the independent variable is taking part in the intervention. (Section 3, 5.2 Confounds: What Weakens the Case; Objective 5.2)

5. What is the key purpose of empowerment evaluation?

Empowerment evaluation is explicitly not an experimental design meant to produce rigorous comparative data. It steps off the ladder of designs because its question is different.
Drawing on Paulo Freire and participatory programming, empowerment evaluation is a collaborative effort in which the community identifies its goals and how to measure them, so it improves its program on its own terms.
Empowerment evaluation often replaces standard validated instruments with measures the community chooses. Its purpose is community-defined improvement, not cutting cost.
Random assignment belongs to the classic experimental design. Empowerment evaluation is a collaborative, participatory approach, not a way of forming control groups.
Correct answer: to help a community improve its program on its own terms. The information needed is the description of empowerment evaluation in 5.1 and the discussion after it. Drawing on Paulo Freire's community theories and participatory programming, empowerment evaluation is a collaborative effort in which the community identifies its goals and how to measure them, often instead of standard validated instruments. The option about rigorous experimental data is the trap: empowerment evaluation is explicitly not an experimental design meant to produce rigorous comparative data. (Section 3, 5.1 Evaluation Designs: Making a Case; Objective 5.1)

6. A planner reviewing a PRECEDE-PROCEED assessment for a youth smoking program asks in what order the intervention is expected to produce change. Which sequence is correct?

This is the order in which the PRECEDE assessments trace causes backward from the health problem. The intervention travels the chain the other way, starting with the factors.
Activities do not change the health problem directly. They target the factors, which should change the risks, and the health problem changes last.
Health problems result from risks, which result from predisposing, enabling, and reinforcing factors. An intervention targets the factors, which should change risks such as smoking, which should then change the health problem.
Administrative and policy resources are the context that helps or hinders implementation, and social conditions come at the end of the chain. This sequence does not follow the logical chain.
Correct answer: factors, then risks, then the health problem. The information needed is the logical chain in 4.1 and Figure 11.3. Health problems result from risks, which result from predisposing, enabling, and reinforcing factors, so an intervention targets the factors, which should change the risks, which should change the health problem. The first option is tempting because it is the order in which the assessments trace causes backward from the problem, but the intervention travels the chain the other way. (Section 3, 4.1 From PRECEDE to a Logical Chain, Figure 11.3; Objective 4.1)

7. A First Nation in the BC Interior, working with the First Nations Health Authority, runs a two-year program in which Elders and youth grow, harvest, and cook traditional foods together. Why would a classic experimental design be a poor fit for evaluating it?

A classic experiment randomly assigns individuals, raising the ethical problem of giving a needed program to only some community members. In a small community controls would likely join informally, which is diffusion of treatment.
Classic experiments do collect baseline data: pre- and posttest or time series data from both the intervention and control groups.
Changes in eating traditional foods can be measured through surveys, records, or community-defined indicators. The poor fit comes from ethics and diffusion, not measurement.
The classic experiment is used for outcome and impact evaluation, the top of the ladder. Process evaluation rests on historical record keeping.
Correct answer: random assignment would withhold the program, and diffusion of treatment would be likely. The information needed is the classic experimental design in 5.1 and diffusion of treatment in Table 11.4. A classic experiment randomly assigns individuals to receive the program or serve as controls, which raises the ethical problem of providing an intervention to only some people in a community that needs it, and in a small community controls would be likely to take part informally, so the comparison would show little difference. The baseline option is wrong because classic experiments collect pre- and posttest data from both groups. (Section 3, 5.1 Evaluation Designs: Making a Case and 5.2 Confounds: What Weakens the Case; Objective 5.1)

8. A new community program has spent its first year recruiting partners and getting sessions running in three neighbourhoods. A board member insists that only a randomized trial of the program's effect on heart disease would be worth doing. Which response best reflects the lesson's view that there is no cookie-cutter standard?

The lesson says results short of a classic experiment are not without merit. Programs run in the real world and success has many levels.
There is no reason to postpone evaluation. When getting a program operating is the achievement, a process evaluation fits now, and outcomes can follow as the program matures.
Qualitative data matter, but the lesson treats quantitative and qualitative data as answering different questions. Neither replaces the other, and counts of delivery are needed here.
Getting a program in place and operating is itself an achievement, so a process evaluation fits now, an outcome evaluation as the program matures, and impacts only when it runs long enough to show them.
Correct answer: a process evaluation now and an outcome evaluation as the program matures. The information needed is 5.3: results short of a classic experiment are not without merit, and when getting a program in place and operating is the achievement, a process evaluation may be all that is needed, with longer-term impacts sought only when the program runs long enough to show them. The qualitative-only option is tempting because qualitative data matter, but the lesson treats quantitative and qualitative data as answering different questions, not as replacements for each other. (Section 3, 5.3 No Cookie-Cutter Standard; Objective 5.3)

✦ Pass the knowledge check with 100% to continue

Section 4

Careers: Putting Theory to Work

⏱ Estimated reading time: 15 minutes

Section 4 of 4

Careers: Putting Theory to Work

Where social and behavioural theory and evaluation are used, and by whom.

The argument

Theory and evaluation: where the money is

Coherence

Theory connected to real circumstances is what makes a program make sense.

Evaluability

Theory says what ought to change, and therefore what to evaluate.

Together they produce the evidence that funders, registries, and communities now expect.

Six paths

Where theory is used

Government or public agencyNonprofit, community-based, schoolsPrivate consultingPrivate industryHealthcare providersAcademic settings

In Canada: PHAC and ministries, health authorities and public health units, charities and community health centres, consultancies, employers and unions, universities.

Two ends

Agency and community organization

Public agency

Adopts a framework, issues the funding announcement, convenes the review panel, oversees through a program officer.

Community organization

Knows the community, writes the application, delivers the program, evaluates or partners to evaluate.

A grant application: problem, theory-justified plan, evaluation plan, staff and resources, experience, budget and timeline.

Three more

Consulting, industry, health care

Consulting

Contracts, project teams, hands-on work with the funder; often the ones writing the evaluation plan.

Industry

Workplace wellness, screening, exercise; associations and unions on policy.

Health care

Information, interpersonal communication and role modelling, feedback from practice.

Academic, and a Canadian addition

Universities and Indigenous health organizations

Academic settings

Generate and test theory; evaluate; train; move research into practice through knowledge translation.

Indigenous health organizations

First Nations Health Authority, national and local Indigenous organizations; community-defined success, OCAP principles.

HEAL connection

Evaluation, careers, and the HEAL map

Outcomes are not endpoints

Judge a program by the part of the map it acts on, and measure upstream outcomes too.

Social construction

Collective behaviour reshapes the context; every career setting takes part.

Carry forward

What to take into the final review

  • Theory lends coherence to a program and tells the evaluator what to measure.
  • Six general settings, plus Indigenous health organizations in Canada; each uses theory in design, implementation, or evaluation.
  • Agencies fund and oversee; community organizations design, deliver, and evaluate; consultants and academics work on both sides.
  • Next: the final assessment, integrating evaluation, logic models, methods, and careers.

Overview

This final section of the course is short and practical. It asks where, in public health work, social and behavioural theory and evaluation are actually used, and by whom. It opens with the argument for why theory has become a qualification in its own right, maps the six general career settings onto Canadian public health, including Indigenous health organizations, and closes with the funding process that connects agencies to community organizations and the grant application that passes between them.

Learning Objectives

  • 6.1 Explain why a working knowledge of social and behavioural theory, together with evaluation skills, has become an important qualification for public health work and for securing program funding.
  • 6.2 Describe the six career settings in which theory is applied, the role theory plays in each, and their Canadian counterparts.
  • 6.3 Describe how theory and evaluation are used in consulting, private industry, health care, and academic settings, including knowledge translation.
  • 6.4 Explain why Indigenous health organizations are a Canadian setting that the six-part typology does not name, and how evaluation is often done there.
  • 7.1 Describe the two ends of the funding process: the public agency that funds and oversees, and the community organization that applies, delivers, and evaluates.
  • 7.2 Outline the six parts of a grant application and explain where theory and evaluation appear in it.

6.1 Theory as a Qualification

The argument of this section is that a good working knowledge of social and behavioural theory and its real-world application has become an increasingly important qualification for public health work, because so much weight now falls on programs that make sense, are effective, and are evidence-based. Theory, properly connected to real circumstances and people, is what lends coherence to an intervention. By this we mean that theory explains why each activity should lead to the change the program wants, so the parts of the program fit together instead of being a list of good ideas. In addition, theory is what allows a meaningful evaluation, because the theoretical base tells you what ought to change and therefore what to evaluate, the point made in Section 1. Put the two together and the argument follows. Theory plus evaluation produces the evidence that improves the knowledge base about what works, and that funders, registries, and communities now expect. Because funders increasingly require that evidence, the person who can supply both halves, a coherent theory-based design and an evaluation that tests it, is the person whose programs get funded. That, to put it bluntly, is where the money is. The argument applies across very different workplaces, which the next subsection maps.

6.2 Six Career Settings

Public health careers where theory matters can be grouped into six settings, each with a characteristic role for theory. By this we mean that the same body of theory is put to different uses depending on what an organization does with programs. The first setting is government or public agencies, domestic and global, where theory is used in program design and management; agencies decide what kind of program to fund and oversee it, rather than delivering it themselves. The second is nonprofit and community-based organizations, and schools, where theory is used in program design and implementation, including evaluation, and in advocacy. The third is private sector consulting organizations, which use theory in program design and evaluation and in technical assistance to the agencies that hire them.

The remaining three settings are private industry, healthcare providers, and academic settings. In private industry, theory guides program design and implementation for workplace health. Healthcare providers use it in program design and implementation alongside their clinical work. Academic settings use it most broadly of all, in behavioural research, program design and evaluation, research collaboration, training, and capacity building. A useful way to tell the settings apart in a case is to ask what the organization does with a program: funds and oversees it, delivers it in the community, designs or evaluates it under contract, runs it for its own workforce, delivers it alongside care, or researches and teaches it. Table 11.5 lists the six settings and the role for theory in each, and adds the Canadian organizations where you would find them.

Table 11.5. Six career settings in public health, the role for theory in each, and where each is found in Canada.

SettingRole for theoryWhere in Canada
Government or public agency, domestic and globalProgram design and managementPublic Health Agency of Canada; Health Canada; Indigenous Services Canada; provincial ministries and agencies such as the BC Centre for Disease Control, Public Health Ontario, and the Institut national de santé publique du Québec; regional health authorities; Ontario's local public health units
Nonprofit and community-based organizations, and schoolsProgram design and implementation (including evaluation); advocacyCanadian Cancer Society, Heart and Stroke Foundation, Canadian Mental Health Association, community health centres, friendship centres, AIDS service organizations, school boards
Private sector consulting organizationsProgram design and evaluation; technical assistanceEvaluation and research consultancies contracted by governments, health authorities, and foundations
Private industryProgram design and implementation (including evaluation) for workplace healthEmployers with extended health benefits and wellness programs; industry associations; labour unions and their health and safety committees
Healthcare providersProgram design and implementation (including evaluation)Hospitals and health authorities, primary care networks and community health centres, pharmacies, public health nursing
Academic settingsBehavioural research, program design, evaluation, research collaboration, training, capacity buildingFaculties of health sciences and public health; research funded by the Canadian Institutes of Health Research and provincial funders such as Michael Smith Health Research BC

Notice that health authorities appear in two rows of Table 11.5: regional health authorities under government or public agencies, and hospitals and health authorities under healthcare providers. This is not an error. In Canada a health authority is a public body that plans and manages public health programs and also delivers health care, so which setting a particular job belongs to depends on the work it involves rather than on the employer's name. For example, a health promotion lead who manages community programs works in the agency role, while a public health nurse delivering services works in the healthcare provider role, even if both are employed by the same health authority.

Interactive 11.4 expands Table 11.5 to eight settings, showing advocacy organizations separately and adding Indigenous health organizations as a Canadian setting. Click each setting in turn to see what people do there, how theory enters the work, and where it happens in Canada; the counter below the panel tells you when you have viewed all eight.

📊 Interactive 11.4: Career explorer.

Click a setting to see what people do there, how theory enters the work, and where it happens in Canada. Eight settings: the six from the table, with advocacy organizations shown separately, plus a Canadian addition.

Select a setting on the left.

0 of 8 settings viewed
Government or public agency

What you do Disseminate and manage funds for a health problem: write funding announcements, support review panels, then oversee the selected organizations as a program officer without delivering the program yourself. Policy offices develop guidelines and regulations and build consensus.

Theory Announcements are increasingly built on a program framework or theoretical approach the agency has adopted. Policy work draws on the ecological model, communications strategies, and organizational mobilization.

In Canada Program consultants and policy analysts at the Public Health Agency of Canada and Health Canada; provincial ministry staff; health promotion leads at regional health authorities; program managers in Ontario public health units.

Nonprofit or community-based organization

What you do Work at the other end of the funding process, in direct contact with the community. Respond to solicitations, or develop a program that meets a need and seek funds for it. Deliver it, then evaluate it or partner with an organization that can. Write grant applications.

Theory The application justifies the program with theory and an analysis of the problem, a little like a PRECEDE-PROCEED analysis, and the evaluation plan follows from it.

In Canada Health promotion coordinators at community health centres, AIDS service organizations, immigrant-serving agencies, friendship centres, and national charities such as the Canadian Mental Health Association.

Advocacy organization

What you do Increase public engagement about an issue and affect policy on it: media advocacy, social marketing and other communications, position papers, help drafting legislation, and the research behind them.

Theory Communications theory and social marketing, with community and organizational change theory; goals are set in terms of behaviour and policy change.

In Canada Policy and communications roles at the Canadian Cancer Society, the Heart and Stroke Foundation, Physicians for a Smoke-Free Canada, and the Canadian Public Health Association.

Private consulting

What you do Carry out work agencies contract to firms with specific expertise: evaluation, program design, organizational development, cultural competency, communications. You work in a project team, the firm competes through proposals, and contracts mean closer contact with the funder's project officer than grants do.

Theory Knowledge of theory and its relationship to design, implementation, and evaluation is very important here, because consultants often write the logic model and evaluation plan.

In Canada Evaluation and research consultancies working for governments, health authorities, and foundations; many hold the Canadian Evaluation Society's Credentialed Evaluator designation.

Private industry and workplaces

What you do Design, manage, and evaluate workplace health promotion: wellness screening and education, exercise programs, facility memberships. Industry associations advocate workplace health policy, and labour unions often run wellness programming and lobby.

Theory Worksite theories (stage-based tailoring, social support, organizational change); evaluation shows the employer or union what the program returns.

In Canada Wellness and occupational health roles at large employers and benefits providers; union health and safety committees; workplace mental health initiatives run with the Canadian Mental Health Association.

Healthcare providers

What you do Beyond care: disseminate health information, provide interpersonal communication and role modelling, and give real-world feedback on which public health approaches work. Providers are also a setting for research on health beliefs, program approaches, and barriers to treatment.

Theory Role modelling and interpersonal communication are Social Cognitive Theory in practice; a pharmacist offering nicotine replacement is a cue to action from the Health Belief Model.

In Canada Health promotion positions in hospitals and health authorities, primary care networks, community health centres, public health nursing, and community pharmacy.

Academic settings

What you do Generate theory through research, evaluate theory-driven interventions, identify useful evaluation methods, and compete for peer-reviewed grants, often with community partners. Share results in journals, briefs, policy recommendations, and with partner communities. Teach and train; consult for agencies.

Theory This is where theory is built, tested, and taught.

In Canada Faculties such as SFU Health Sciences; research funded by the Canadian Institutes of Health Research, which uses the term knowledge translation for moving findings into practice.

Indigenous health organizations (a Canadian addition)

What you do First Nations, Inuit, and Métis organizations design, deliver, and evaluate health promotion in their own communities and at regional and national levels. The First Nations Health Authority in British Columbia took over federal First Nations health programs in the province in 2013; Inuit Tapiriit Kanatami, the National Collaborating Centre for Indigenous Health, and local friendship centres also employ health promotion and evaluation staff.

Theory Community and cultural theories sit alongside Indigenous knowledge; evaluation often follows the empowerment model, with the community defining success and controlling its data under OCAP principles.

In Canada Wellness and evaluation roles with the First Nations Health Authority, Métis and Inuit organizations, tribal councils, and friendship centres.

Across the eight panels in Interactive 11.4 the pattern is the same: the job titles change, but each setting uses theory to design, deliver, or evaluate a program. Advocacy organizations, a variant of the nonprofit, focus on increasing public engagement about an issue and affecting public policy on it, through media advocacy, social marketing and other communications, position papers, and help drafting legislation, drawing on communications theory, social marketing, and community and organizational change theory. The difference from other nonprofits lies in the target. A community-based organization usually tries to change what individuals do, whereas an advocacy organization usually tries to change the conditions around them, such as a law, a regulation, or a funding decision. For that reason its goals are set in terms of behaviour and policy change, and its evaluation asks whether the policy moved as well as whether the message reached people. The next subsection looks more closely at the settings that work alongside government and community organizations.

6.3 Consulting, Industry, Health Care, and Academic Settings

6.3.1 Consulting, industry, and health care

More and more of the work of government agencies is carried out by private consulting organizations with specific expertise: evaluation, program design, organizational development, cultural competency, communications campaigns, materials, information technology. In a consulting firm you work in a project team, and the firm competes for work through proposals much like grant applications. However, the arrangement that follows is different from a grant. Consultants usually work under contracts rather than grants, which means more hands-on interaction with the funding agency's project officer. By this we mean that under a contract the agency is buying a specific piece of work it has defined, so it stays closely involved, whereas a grant funds an organization to carry out a program it has proposed itself. Knowledge of theory and its relationship to design, implementation, and evaluation is very important here, because consultants are often the ones writing the logic model and the evaluation plan.

Private industry is a different kind of setting, because here the organization runs programs for its own workforce rather than for a community. Private businesses that provide health insurance may run workplace health promotion to manage costs, through wellness screening and education, exercise programs, health facility memberships, and sometimes in-house staff. The cost motive explains the shape these programs take. An employer that pays for health benefits, and that loses productivity when workers are ill, has a direct financial interest in its employees' health, so it tends to judge a wellness program by what it returns in fewer claims and sick days. This is also why evaluation matters in the workplace: the employer, or the union, wants evidence that the program is worth what it costs. In addition to employers, two other players work on workplace health in this setting: industry associations advocate workplace health policy, and labour unions often handle wellness programming and lobby as well.

Healthcare providers, public or private, large or small, are a setting in which health promotion sits alongside clinical care. They have three key roles beyond care. The first is disseminating health information. The second is providing interpersonal communication and role modelling, which is how a trusted clinician can influence behaviour. The third is giving real-world feedback on which public health approaches work, because providers see the results in their patients. For example, a pharmacist who offers nicotine replacement to a customer who smokes is disseminating information and acting as a cue to action at the same time. Healthcare providers are also a setting for research on health beliefs and barriers to treatment.

Worked Example 11.6 places a new graduate in a unionized workplace, where management and the union disagree about the cause of a problem. Watch how theory is used to give both proposals a place in one program, and how the evaluation has to answer each side's question.

Worked Example 11.6: A Wellness Program in a Unionized Workplace

The case. A large Canadian employer with a unionized workforce wants to reduce sick days and improve mental health. Management proposes an app-based wellness challenge. The union's health and safety committee argues that shift scheduling and workload are the real problem and wants organizational change. A health promotion graduate has been hired to design something both sides will accept and to evaluate it.

Step 1: Identify the setting and the players. This is the private industry setting: an employer running workplace health promotion, partly to manage costs, and a labour union whose health and safety committee works on wellness and workplace policy.

Step 2: Use theory to frame the problem at more than one level. The ecological perspective holds that behaviour is shaped at several levels at once, and worksite theories add organizational change and social support. The app-based challenge works at the individual level, while scheduling and workload are organizational. A multilevel program can include both: a voluntary wellness challenge alongside a joint management and union review of shift scheduling and workload.

Step 3: Plan the process evaluation. For both sides it must show that each component was delivered as designed: how many employees used the app and how often (dosage), and whether the scheduling review met, made recommendations, and saw them put into practice.

Step 4: Plan the outcome evaluation. The employer will look for short-term change in sick days and in participation. The union will look for change in workers' reports of workload and stress. Both need the same measures taken at baseline and at follow-up.

What this shows. Theory does not pick a side; it gives each proposal a place in one model, and the evaluation then answers the question each party cares about. A common mistake is to evaluate only the component that the party paying for the program proposed.

Consulting firms, employers, and healthcare providers mostly apply theory. The last of the six settings is where theory is built and tested.

6.3.2 Academic settings and knowledge translation

Universities generate theory through research, evaluate theory-driven interventions, and identify useful evaluation methods, all of which means competing for research grants through peer-reviewed applications, often with community partners. The difference from the settings above is that academic work aims to add to general knowledge about what works, not only to deliver or evaluate one program. However, that knowledge is useful only if it reaches the people who run programs. For that reason results are shared in journals and at conferences, and also in research briefs, policy recommendations, and with the partnering communities, which is how research gets transferred to practice. Canadian funders call this knowledge translation. By this we mean the deliberate work of moving research findings into the hands of practitioners, policymakers, and communities in a form they can use. In addition to research, academics teach students and practitioners and advise agencies. The possibilities are many and can only increase. The six settings, however, come from a typology written for the United States, and the next subsection shows what it leaves out in Canada.

6.4 Four People, and a Canadian Addition

Four hypothetical profiles, none of them real cases, illustrate the settings. Three are set in the United States, where this typology of settings originated: a program officer at a federal agency, a communications associate at an advocacy nonprofit, and a research director at a consulting firm. The fourth is set in a Canadian setting the typology does not cover, an evaluation specialist with a First Nations health organization. Each card gives the person's training, work, and view of theory.

Government:
program officer

Program officer at the United States Department of Health and Human Services. Master’s in public health with a health promotion concentration, bachelor’s in social psychology, five years at an urban nonprofit running cardiovascular, diabetes, and HIV education, then five years managing community programs at a state health department. She helps design, implement, and evaluate a national grant program to prevent obesity and diabetes among youth and families in urban areas with few recreation facilities. Funded projects combine education about risk with environmental change, through community mobilization and public-private partnerships. She monitors grants in her region, gives technical support, prepares reports, and works with an evaluation officer to make sure every grant is evaluating well. Her lesson from community work: many things connect to any given health problem, so the program attacks obesity from several directions.

Advocacy:
communications

Communications associate at Access for All, an advocacy nonprofit working to increase access to prevention services for underserved populations. Master’s in public health with an emphasis on communications. Years with small community-based organizations in Nogales, Arizona, on the border, and in Dade County, Florida, serving migrant agricultural workers and Haitian immigrants, where she learned web design, event planning, and the value of evaluation data in winning new funding. She now develops communications strategies aimed at legislators, officials, and minority community organizations, and produces web, social media, and press content, setting communication goals in terms of behaviour and policy change. Her view of her job: local people know the facts on the ground; she translates that for researchers in the language of theory and for policymakers in the language of achievable goals.

Consulting:
research director

Research director of social and health programs at Advantage International, a private consulting firm in global health. Doctorate in public health and a master’s in international development and communications. Peace Corps volunteer in the Dominican Republic on cholera and waterborne disease communication, two years teaching English in Chile, then eight years at a social marketing company doing research and communications on HIV, immunization, and family health for the United States Agency for International Development and the World Bank. He now leads proposal development, keeps track of current theory and research, directs projects with strong research components (for example, attitudes and risk behaviour among migrant farmworkers in South Africa), leads each project’s evaluation team, and writes reports, articles, and presentations. His quote: people sometimes roll their eyes at theory, but he uses it all the time, because how else would you connect the reality you see to what to do about it?

Indigenous health:
evaluation specialist

Evaluation specialist with a First Nations health organization in British Columbia. Bachelor’s in health sciences from SFU, then a master’s in public health with a practicum at a regional health authority. She works with communities that are designing wellness programs, from land-based youth camps to diabetes prevention, helping each community write its own logic model, choose indicators it considers meaningful, and collect baseline and follow-up data it owns under OCAP principles. She also prepares the reports funders require, translating community-defined success into the process and outcome language the funder expects. Her view: the theory she learned tells her what a program is trying to change; the community tells her what change is worth measuring, and the job is to hold both.

The four profiles share a habit. Each person uses theory to connect what they see on the ground to what to do about it, and each works with evaluation, whether monitoring grantees, using evaluation data to win funding, leading an evaluation team, or helping communities choose their own indicators.

Canada adds a setting that the six-part typology does not name: Indigenous health organizations. The First Nations Health Authority in British Columbia took over federal First Nations health programs in the province in 2013 and has planned and delivered them since, and national and local Indigenous organizations, such as Inuit Tapiriit Kanatami, the National Collaborating Centre for Indigenous Health, and friendship centres, employ health promotion and evaluation staff. Their evaluation work often follows the empowerment model from Section 3, with communities defining success and controlling their own data under the First Nations principles of ownership, control, access, and possession (OCAP). By this we mean that a First Nations community, rather than an outside funder or researcher, owns the information collected about it, decides how that information is gathered and used, and decides who may see it. This changes the evaluator's role. Instead of arriving with a fixed set of indicators and taking the data away, the evaluator helps the community choose what counts as success and then collects evidence that stays under the community's control. For example, a community running a land-based youth camp might decide that young people's connection to language and land is the change worth measuring, alongside any outcome the funder requires.

A caution about the typology

The six-setting typology was written for the United States, where private insurance drives workplace programs and federal grants shape the nonprofit sector. In Canada, ministries, health authorities, and public health units deliver much of what a United States nonprofit might do under a grant, and Indigenous self-determination in health has created organizations with no counterpart in that typology. The roles for theory and evaluation are the same. The organizations that hold them are not.

Worked Example 11.7 asks a graduating student to compare three Canadian job postings using the settings in this section. Watch how the setting decides which type of evaluation fills the working week, and who gets to define success.

Worked Example 11.7: Three Job Postings

The case. A graduating SFU health sciences student is weighing three job postings: a health promotion coordinator at a regional health authority in the BC Interior, developing and evaluating community programs on physical activity and healthy eating; an evaluation and quality improvement analyst with an Indigenous health organization, supporting communities that design their own wellness programs; and a research coordinator on a university trial, funded by the Canadian Institutes of Health Research, testing a theory-based intervention to increase HPV vaccination among young adults.

Step 1: Place each posting in a setting. The coordinator is closest to the government or public agency setting as Canada organizes it, since Table 11.5 lists regional health authorities there. The analyst works in the Canadian setting the typology does not name, an Indigenous health organization. The research coordinator is in an academic setting.

Step 2: Describe how theory shows up in a typical week. The coordinator chooses and adapts theory-based programs, for example social cognitive or ecological approaches to physical activity, and checks that the activities fit the factors identified in assessment. The analyst helps communities write their own logic models, drawing on community and cultural theories alongside Indigenous knowledge. The research coordinator delivers an intervention whose components come from a named theory and keeps its delivery faithful to the design.

Step 3: Identify the main type of evaluation. The coordinator spends most time on process and outcome evaluation of community programs. The analyst supports empowerment evaluation, translating community-defined indicators into the process and outcome reports funders expect. The research coordinator works on an outcome evaluation built into an experimental design, where process records of fidelity protect the test.

Step 4: Answer the final question. The university trial would let the student test a theory, because it is designed to show whether the theory-based intervention changes vaccination. The Indigenous health organization would let a community test its own definition of success.

What this shows. The same knowledge of theory and evaluation travels across settings. What changes from one setting to the next is who sets the goals and which question the evaluation is built to answer.

Every setting in this section sits somewhere in a funding relationship. The last two subsections describe that relationship from both ends.

7.1 Two Ends of the Funding Process

The first two settings in Table 11.5 are two ends of one process. By this we mean that the money for most health promotion programs passes from one kind of organization, which funds and oversees, to another, which delivers, and theory and evaluation travel with it in both directions. At the funding end, a government agency, federal, provincial or state, local, or international, disseminates and manages funds allocated to a health problem. Its funding announcements describe the purpose, the activities requested, the funds available, and the criteria for success, and are increasingly based on a program framework or theoretical approach the agency has adopted. This is the first point at which theory enters the funding process: before any community organization applies, the agency may already have decided which approach the programs it funds should follow.

The people who make those decisions are program heads and program officers. They decide what will be funded but do not carry out the program; the organizations selected do, overseen by a program officer. This division of labour is easy to miss. For example, a program officer monitors grantees, offers technical support, and reads their reports, but does not run the sessions in the community. To choose among applicants, a panel of experts and agency staff rates applications on the best plan at the best price with adequate organizational capability. Agencies also do policy work, issuing guidelines and regulations (air and water pollution, secondhand smoke) and building consensus on recommended policies. Figure 11.4 shows the two ends of the funding process and what passes between them.

Public agency adopts a program framework issues a funding announcement review panel rates applications program officer oversees Community organization knows the community writes the application designs and delivers the program evaluates, or partners to evaluate funds, framework, success criteria reports, evaluation data, evidence Consultants and academics work on either side Private foundations act like agencies at the funding end

Figure 11.4. Funds, a program framework, and success criteria flow from a public agency to a community organization, and reports, evaluation data, and evidence flow back; private foundations act like agencies at the funding end, and consultants and academics work on either side.

Community-based organizations are at the other end, in direct contact with people and with intimate knowledge of the situations surrounding health behaviour. They write the applications, deliver the programs, and evaluate them or partner with someone who can. Here theory is used in program design and evaluation, and it can enter in two ways. In the first, the organization responds to a solicitation, so its program must fit the framework the agency has already announced. In the second, the organization has developed a program it believes meets a need and must seek funding for it, so it has to justify its own choice of theory. In both cases the evaluation results flow back to the funder as reports and evidence, which closes the loop shown in Figure 11.4. Not every funder is a government agency, however. The exception is the private foundation, a nonprofit that acts like a public agency by disseminating funds; the Ford Foundation and the Robert Wood Johnson Foundation are United States examples, and Canada's community foundations, such as the Vancouver Foundation, play a similar role. A task common to all these organizations is writing grant applications, the subject of the final subsection.

7.2 The Grant Application

A grant application typically contains six parts: a description of the health problem to be addressed; a plan to address the problem, justified by theory and an analysis of the problem; a plan to evaluate the program's effectiveness; staff and agency resources, with their qualifications and capabilities; the organization's experience doing this kind of work; and a budget and a timeline for implementation. The first three parts carry the argument of the application, while the last three show that the organization can carry it out. Theory appears in the plan, the second part, where it justifies why the chosen activities should change the factors behind the problem. Evaluation appears as its own part, the third, and it follows from the plan because the plan says what ought to change. For example, if the plan rests on Social Cognitive Theory and promises to raise self-efficacy, the evaluation plan should measure self-efficacy at baseline and follow-up; an evaluation plan that measured something else would not test the program the application describes. Each part is described below with how it relates to the logic model, to theory, or to evaluation.

1. A description of the health problem to be addressed

The epidemiological and social assessment in miniature: who is affected, how badly, and in what context. In logic model terms, the problem column.

2. A plan to address the problem, justified by theory and an analysis of the problem

This is a little like a PRECEDE-PROCEED analysis: the factors identified, the theory that explains them, and the activities chosen to change them. This is where a reviewer decides whether the program makes sense.

3. A plan to evaluate the effectiveness of the program

Process, outcome, and (if realistic) impact questions, the design, the indicators, and the baseline plan. Because part 2 says what ought to change, part 3 follows from it. A reviewer reads the two together and checks that the evaluation measures the changes the plan predicted, rather than something easier to count.

4. Staff and agency resources, with their qualifications and capabilities

The inputs column of the logic model, and the reviewer's basis for judging organizational capability.

5. The organization's experience doing this kind of work

Track record: previous programs, previous evaluations, and relationships with the community. The Winnipeg director in Worked Example 11.1 needed stronger material here.

6. A budget and a timeline for implementation

What it will cost and when each output will be delivered. Reviewers rate applications partly on price, and the timeline becomes the yardstick for process evaluation. For example, if the timeline promises six workshops by the end of the first year, the process evaluation will later ask whether six workshops were in fact held by then.

Read as a whole, a grant application is a logic model in prose: the problem column, a theory-based chain from activities to outcomes, the evaluation that will test it, the inputs, and a timeline against which process evaluation will judge delivery.

One closing observation applies to every setting, profile, and case in this section. Every setting rewards the same combination: theory connected to real circumstances, a program built from it, and an evaluation that checks whether the predicted changes occurred. That combination is what this course has been assembling throughout, and it is the qualification public health increasingly asks for. This section has described six career settings and their Canadian counterparts, the funding process from both ends, and the parts of a grant application in which theory and evaluation appear. The key takeaways below draw the section together, and the knowledge check prepares you for the lesson's final review and assessment.

Key Takeaways

  • 6.1 A good working knowledge of social and behavioural theory has become an increasingly important qualification for public health work, because so much weight now falls on programs that make sense, are effective, and are evidence-based. Theory, properly connected to real circumstances and people, lends coherence to an intervention, and it makes a meaningful evaluation possible because it says what ought to change. Theory plus evaluation produces the evidence that improves the knowledge base about what works, which is why the lesson says, bluntly, that this is where the money is. If you are asked to justify the claim, give both halves: coherence from theory, and evidence from evaluating what the theory predicted.
  • 6.2 Theory is applied in six general settings: government or public agencies (program design and management), nonprofit and community-based organizations and schools (design, implementation including evaluation, and advocacy), private consulting (design, evaluation, technical assistance), private industry (workplace health programs), healthcare providers (design and implementation), and academic settings (research, evaluation, training, capacity building). In Canada these correspond to bodies such as the Public Health Agency of Canada, Health Canada, provincial ministries and agencies, regional health authorities and public health units, charities and community health centres, evaluation consultancies, employers and unions, hospitals and primary care, and universities funded by the Canadian Institutes of Health Research. Advocacy organizations are a nonprofit variant focused on public engagement and policy. To classify a job in an exam question, look at what the organization does with programs: funds and oversees them, delivers them, evaluates them under contract, runs them for its own workforce, or researches them.
  • 6.3 Consulting firms carry out work that agencies contract out, such as evaluation, program design, and communications, in project teams that compete through proposals; they usually work under contracts, which means closer contact with the funder's project officer than a grant involves. Private industry runs workplace health promotion, often to manage insurance costs, while industry associations and labour unions work on workplace health policy and wellness programming. Healthcare providers have three roles beyond care: disseminating health information, interpersonal communication and role modelling, and real-world feedback on which approaches work. Academic settings generate and test theory and evaluation methods with peer-reviewed grants and move findings into practice through briefs, policy recommendations, and partner communities, which Canadian funders call knowledge translation. Keep contracts and grants apart: consultants usually hold contracts, while community organizations and academics usually hold grants.
  • 6.4 Canada adds a setting the six-part typology does not name: Indigenous health organizations, such as the First Nations Health Authority in British Columbia, which took over federal First Nations health programs in the province in 2013. The typology was written for the United States, where private insurance drives workplace programs and federal grants shape the nonprofit sector; in Canada, public bodies deliver much of that work, and Indigenous self-determination in health has created organizations with no counterpart in the typology. Evaluation in these organizations often follows the empowerment model, with communities defining success and controlling their data under OCAP principles, as Dana Wilson's profile shows. The roles for theory and evaluation are the same everywhere, so name the setting before you describe the role.
  • 7.1 The government agency and the community organization are two ends of one funding process. The agency disseminates and manages funds for a health problem, increasingly builds its funding announcement on a program framework or theoretical approach it has adopted, has a review panel rate the applications, and oversees the selected organizations through a program officer who does not deliver the program. The community organization, in direct contact with the people served, writes the application, delivers the program, and evaluates it or partners with someone who can, sending reports and evidence back, as Figure 11.4 shows. Private foundations, such as the Vancouver Foundation, act like agencies at the funding end. A typical exam trap is to give the program officer delivery tasks; officers decide frameworks and oversee, while grantees deliver.
  • 7.2 A grant application typically has six parts: the health problem, a plan justified by theory and an analysis of the problem (a little like a PRECEDE-PROCEED analysis), an evaluation plan, staff and agency resources with their qualifications, the organization's experience, and a budget and timeline. Theory lives in the plan, and evaluation has its own part that follows from the plan, because the plan says what ought to change. Several parts map onto the logic model: the problem column, the inputs column in staff and resources, and the timeline as the yardstick for process evaluation. If asked where theory appears in an application, name the plan; if asked where evaluation appears, name the evaluation plan and explain that it follows from the theory.
Knowledge Check: this section

1. A program officer at a federal agency manages a set of grants for youth obesity prevention. Which of the following does she NOT do?

Program heads and officers do help decide the framework or theoretical approach on which funding announcements are increasingly based, so this is part of her role.
Monitoring grantees and offering technical support is the program officer's core oversight work, so she does do this.
Making sure grantees evaluate well is part of overseeing them, so working with an evaluation officer on grantees' evaluations is within her role.
Program officers decide what will be funded and oversee it, but they do not carry out the program. Delivering sessions to youth is the work of the organizations that won the grants.
Correct answer: deliver the program's education sessions. The information needed is the description of the funding process in 7.1 and the program officer's profile in 6.4. Program heads and officers help decide the framework, monitor grants, provide technical support, and make sure grantees evaluate well, but they do not carry out the program; delivery is the work of the organizations that won the grants. Working with an evaluation officer is tempting as the odd one out, but monitoring grantees' evaluations is part of overseeing them. (Section 4, 7.1 Two Ends of the Funding Process; Objective 7.1)

2. Which career setting usually works on a contract basis, with more hands-on interaction with the funding agency's project officer than a grant involves?

Academics usually compete for peer-reviewed research grants, not contracts, so this arrangement does not describe them.
Community-based organizations also apply for funding, but usually through grants for programs they propose, not contracts with closer project-officer involvement.
Consulting firms carry out work agencies contract to firms with specific expertise. Under a contract the agency is buying work it defined, so there is more hands-on interaction with its project officer than under a grant.
Healthcare providers deliver care and health promotion alongside it. The lesson does not describe them as working mainly under contracts to funding agencies.
Correct answer: private sector consulting organizations. The information needed is 6.3.1. Consulting firms carry out work that agencies contract to firms with specific expertise, compete through proposals, and usually work under contracts rather than grants, which means more hands-on interaction with the funding agency's project officer. Community-based organizations are tempting because they also apply for funding, but they usually do so through grants, as do academics. (Section 4, 6.3 Consulting, Industry, Health Care, and Academic Settings; Objective 6.3)

3. A nonprofit spends most of its effort on media advocacy, social marketing, drafting position papers, and helping to draft legislation on sugary drink taxation. What kind of organization is it, and which theories would its staff draw on most?

A private foundation is a nonprofit whose distinguishing role is disseminating funds. The Health Belief Model and Transtheoretical Model are individual-level theories, not the communications and change theories advocacy uses.
Advocacy organizations are nonprofits focused on public engagement and policy, using media advocacy, social marketing, position papers, and legislative drafting. Their staff draw on communications theory, social marketing, and community and organizational change.
Healthcare providers disseminate information and model behaviour alongside care. Media advocacy and drafting legislation are the work of advocacy organizations.
Consulting firms carry out contracted work such as evaluation and program design, often writing logic models. Media advocacy and policy work describe an advocacy organization.
Correct answer: an advocacy organization, drawing on communications theory, social marketing, and community and organizational change. The information needed is the description of advocacy organizations in 6.2 and the advocacy panel of Interactive 11.4. Advocacy organizations are nonprofits focused on public engagement and policy, working through media advocacy, social marketing and other communications, position papers, and help drafting legislation. A private foundation is tempting because it is also a nonprofit, but a foundation's distinguishing role is disseminating funds, as 7.1 explains. (Section 4, 6.2 Six Career Settings, Interactive 11.4; Objective 6.2)

4. Sharing research results through research briefs, policy recommendations, and with partnering communities, and not only in journals, is important for which reason?

Results shared through research briefs, policy recommendations, and partner communities are how research moves into practice, which Canadian funders call knowledge translation.
The Government Performance and Results Act concerns performance monitoring and accountability for public funds, not how academics share research findings.
Research grants and journal publication still go through peer review. Wider sharing adds to publication; it does not replace or avoid review.
Sharing results depends on evaluation, since evaluated interventions produce the findings to share. It does not replace the need to evaluate programs.
Correct answer: it is how research gets transferred to practice. The information needed is 6.3.2. Results shared through research briefs, policy recommendations, and with partner communities are how research moves into practice, which Canadian funders call knowledge translation. The option about avoiding peer review is wrong because research grants and journal publication still go through peer review; wider sharing adds to publication rather than replacing it. (Section 4, 6.3 Consulting, Industry, Health Care, and Academic Settings; Objective 6.3)

5. Beyond providing care, which set of roles do healthcare providers play in health promotion?

Issuing regulations, building policy consensus, and managing grant programs are the tasks of government or public agencies, not of healthcare providers in their health promotion role.
Writing funding announcements and convening review panels belong to public agencies at the funding end of the process, not to healthcare providers.
Beyond care, the lesson names three roles: disseminating health information, interpersonal communication and role modelling, and real-world feedback on which public health approaches work, because providers see results in their patients.
Lobbying on workplace health policy belongs to industry associations and labour unions in the private industry setting, not to healthcare providers.
Correct answer: disseminating information, interpersonal communication and role modelling, and real-world feedback. The information needed is 6.3.1, which names these three roles beyond care. Issuing regulations and managing grant programs are tempting because hospitals are often public bodies, but those are tasks of public agencies, as are writing funding announcements and convening review panels (7.1); lobbying on workplace policy belongs to industry associations and unions. (Section 4, 6.3 Consulting, Industry, Health Care, and Academic Settings; Objective 6.3)

6. A coordinator at a community health centre spends Monday justifying a peer-education program with Social Cognitive Theory for a grant application. Which part of the application is she writing, and which part should follow directly from it?

The budget and timeline and the staff and agency resources show the organization can carry out the work. Justifying a program with theory belongs to the plan, not to either of these parts.
The description of the health problem says who is affected and how badly. Justifying the program with Social Cognitive Theory is the plan, and the evaluation plan is what follows from it.
This reverses the relationship. The evaluation plan is derived from the theory-based plan, because the plan says what ought to change and therefore what to measure.
Theory justifies the plan to address the problem, a little like a PRECEDE-PROCEED analysis. The evaluation plan follows directly, because the plan says what ought to change and so what to measure.
Correct answer: the theory-justified plan, followed by the evaluation plan. The information needed is the six parts of a grant application in 7.2. Theory justifies the plan to address the problem, a little like a PRECEDE-PROCEED analysis, and the evaluation plan follows from it because the plan says what ought to change and therefore what to measure. The reversed option is tempting because both parts are named, but the evaluation is derived from the theory-based plan, not the other way around. (Section 4, 7.2 The Grant Application; Objective 7.2)

7. A hiring manager asks why the lesson calls the combination of theory and evaluation "where the money is." Which answer gives the lesson's reasoning?

Theory connected to real circumstances lends coherence and names what ought to change; evaluation checks whether it changed. Together they produce the evidence that funders, registries, and communities now expect, which is where the money is.
The lesson's claim is about coherence and evidence, not cost. Nothing says theory-based programs are always cheaper to run.
No funder pays only for one theory. The argument applies to any theory properly connected to real circumstances, and the lesson discusses several.
Evaluation depends on theory, which tells it what to measure. Without theory an evaluation has nothing specific to check except whether the program happened.
Correct answer: coherence from theory plus evidence from evaluation. The information needed is the argument in 6.1. Theory lends coherence to an intervention and makes meaningful evaluation possible by naming what ought to change; evaluation then checks the change; and the two together produce the evidence that improves the knowledge base and that funders, registries, and communities expect. The claim is not about cost or about any one theory, and the option that evaluation can replace theory misses the point of Section 1 that an evaluation without theory has nothing specific to measure. (Section 4, 6.1 Theory as a Qualification; Objective 6.1)

8. A graduate joins a First Nations health organization in British Columbia as an evaluation specialist, helping communities write their own logic models and choose indicators they consider meaningful. Why does the lesson describe this as a Canadian setting that the six-part typology does not name?

Evaluation is done in Indigenous health organizations, often following the empowerment model. The lesson says the roles for theory and evaluation are the same; only the organizations differ.
The six-setting typology was written for the United States. Indigenous self-determination in health in Canada, as with the First Nations Health Authority, has created organizations with no counterpart in it.
Indigenous health organizations often use empowerment evaluation, with community-defined success and data controlled under OCAP principles, not classic experiments.
Consulting firms do contracted work for agencies. Indigenous health organizations design, deliver, and evaluate health promotion in their own communities, which is why the lesson treats them as a separate setting.
Correct answer: the typology was written for the United States, and Indigenous self-determination has created organizations it does not include. The information needed is 6.4 and its callout on the typology. Organizations such as the First Nations Health Authority design, deliver, and evaluate health promotion in their own communities, often using the empowerment model with community-defined success and data controlled under OCAP principles, as Dana Wilson's profile shows. The option saying evaluation is not done there is wrong, because the roles for theory and evaluation are the same as elsewhere; only the organizations differ. (Section 4, 6.4 Four People, and a Canadian Addition; Objective 6.4)

✦ Pass the knowledge check with 100% to continue

Section 5

Final Review & Assessment

⏱ Estimated time: 25 minutes

Bringing It All Together

This lesson closed the loop the course has been drawing since its first pages. Evaluation, in plain terms, asks whether you did what you proposed, whether the program had an effect on what it set out to achieve, and whether the program model and theory were useful, with cost and community goals sometimes added. The funding environment now demands it: the accountability movement that followed the Government Performance and Results Act and the Program Assessment Rating Tool, and its Canadian counterparts in Treasury Board results policies and provincial standards, made stated goals the measure of success, and the move to evidence-based practice made evaluation data the evidence that registries compile and funders require. The four reasons to evaluate, accountability, learning and improvement, theory, and efficiency, each shape what gets collected. And theory is what makes an evaluation specific, because a theory-based program predicts particular changes that an evaluation can check.

The three basic types of evaluation ask three questions. Process evaluation asks whether the components were implemented as planned, and supplies fidelity and dosage data that separate a failed theory from a failed delivery. Outcome evaluation asks what short-term effect the program had, measured from baseline to follow-up on the factors theory targeted. Impact evaluation asks whether the health problem itself changed, which needs an extended program or a followed cohort. The logic model gives each its column: inputs, the problem, outputs, short-term outcomes, long-term impacts, and the indicators for each, all descending from the logical chain of PRECEDE assessments. Making a case that the program caused the change is like being a lawyer: designs climb from record keeping through qualitative approaches, periodic inventories, and benchmarking to quasi-experimental and classic experimental designs, and each must anticipate confounds such as history, maturation, testing, regression to the mean, selection bias, attrition, and diffusion of treatment. There is no cookie-cutter standard; success has many levels, quantitative and qualitative data answer different questions, and empowerment evaluation lets a community define success on its own terms.

The final section asked where all of this is practised. The answer was six settings, government and public agencies, nonprofit and community-based organizations, private consulting, private industry, healthcare providers, and academic settings, each using theory in program design, implementation, or evaluation, and each mapped in this lesson onto Canadian institutions from the Public Health Agency of Canada to regional health authorities, community health centres, unions, universities, and Indigenous health organizations such as the First Nations Health Authority. The closing argument of that section is the course's argument: theory connected to real circumstances gives a program coherence and tells the evaluator what to measure, and theory plus evaluation produces the evidence public health now runs on.

Key Takeaways from this lesson

  1. You can explain evaluation in plain terms as a short list of questions: did we do what we proposed, did the program have an effect, was the theory useful, and sometimes what did it cost and were community goals met. You can also explain why funders now require it, through the accountability movement (GPRA and PART in the United States, Treasury Board results policies and provincial standards in Canada) and the move to evidence-based practice, with its registries and model programs. Remember that a goal stated in a planning document becomes the measure of success, and that "evidence-based" refers to rated evidence, not only to randomized trials.
  2. You can name the four reasons to evaluate, accountability, learning and improvement, theory, and efficiency, and spot which one drives a request by asking who is asking and why. You can explain that a theory-based program is a set of predictions, so theory tells the evaluator what ought to change and therefore what to measure. The practical reminder: without baseline and follow-up measures of what the theory predicted, a program can show only that it happened.
  3. You can distinguish process evaluation (implemented as planned, with fidelity and dosage data), outcome evaluation (short-term change from baseline to follow-up), and impact evaluation (the health problem itself, over years or in a followed cohort), and you know that process data separate a failed theory from a failed delivery. Always check a report's definitions, because PRECEDE-PROCEED swaps the labels outcome and impact.
  4. You can trace the logical chain of the PRECEDE assessments, from factors to risks to the health problem, and build a logic model with inputs, the problem, outputs, short-term outcomes, long-term impacts, and indicators and measures, each column tested by its own type of evaluation. The most useful habit is to keep inputs (what is invested) and outputs (what is delivered) apart.
  5. You can place an evaluation design on the ladder from record keeping to the classic experiment by asking whether there is a comparison group and whether it was randomly assigned, and you can name the seven confounds that weaken a causal claim. You also know that there is no cookie-cutter standard and that empowerment evaluation lets a community define success on its own terms. Fit the design to the program, and limit the claim to what the design can support.
  6. You can describe the six settings where theory is applied, government and public agencies, nonprofit and community-based organizations, consulting, private industry, health care, and academic settings, with their Canadian counterparts and with Indigenous health organizations as a Canadian addition. Every setting rewards the same combination of theory connected to real circumstances and an evaluation that checks what the theory predicted, so name the setting first and then describe how theory enters the work there.
  7. You can describe the funding process from both ends: the agency that adopts a framework, issues the funding announcement, and oversees grantees through a program officer, and the community organization that applies, delivers, and evaluates. You can list the six parts of a grant application and explain that theory justifies the plan and the evaluation plan follows from it. Remember that program officers oversee programs but do not deliver them.

Final Knowledge Assessment

This assessment covers all material from this lesson. Its last five questions ask you to integrate the lesson in one Canadian scenario. A provincial health authority has funded a community organization for three years to deliver a program that aims to increase physical activity among newcomer families in a mid-sized city, using Social Cognitive Theory (family role models, group walks, skill building) and social network strategies (recruitment through settlement agencies and faith communities). Questions 16 to 20 follow that program from the reasons to evaluate, through the evaluation questions, the logic model, and the design, to the people who would do the work. You must score 100% to complete the lesson. Review the feedback for any incorrect answers and try again.

Final Assessment: Evaluation and Careers in Health Promotion (20 Questions)

1. A community organization keeps attendance sheets and teacher thank-you notes but has no baseline or follow-up data. Which of the basic evaluation questions can it answer, and which can it not?

Without baseline and follow-up data the organization cannot show an effect. Attendance sheets show only that activities happened, which answers whether it did what it proposed.
Attendance records show the planned activities took place, the first evaluation question. Effect and the usefulness of the theory need measures of predicted change taken before and after the program.
Judging whether the theory was useful needs measures of each predicted link before and after. Attendance records can show delivery but not whether the theory held.
The records are real data, but data about delivery are not data about change. Only the first of the three questions can be answered from attendance.
Correct answer: it can answer whether it did what it proposed, but not whether the program had an effect or the theory was useful. The information needed is the plain-terms definition in 1.1 and Worked Example 11.1. Attendance records show that the planned activities happened, the first evaluation question; the second and third need measures of what the theory said would change, taken before and after the program. The option claiming that all three can be answered is tempting because the records are real data, but data about delivery are not data about change. (Section 1, 1.1 Evaluation Without the Jargon and Worked Example 11.1; Objectives 1.1 and 2.2)

2. Which development can be described as an even more profound change than the accountability movement?

The Government Performance and Results Act is part of the accountability movement itself, the change being compared, not something more profound than it.
Strategic planning documents spread as a consequence of the accountability movement. They are part of that change, not a more profound one.
The Program Assessment Rating Tool followed GPRA as part of the accountability movement, so it cannot be the more profound change.
The move to evidence-based practice, modelled on evidence-based standards of care in medicine, is described as arguably more profound than accountability. It makes evaluation data the evidence that registries rate and funders require.
Correct answer: the move to evidence-based practice. The information needed is 1.3, which describes the move to evidence-based practice, modelled on evidence-based standards of care in medicine, as arguably an even more profound change than accountability. It makes evaluation data the evidence, leads to rated registries of interventions, and drives funders to require model programs. GPRA, PART, and strategic planning documents are tempting, but they are the accountability movement itself, the change being compared. (Section 1, 1.3 Evidence-Based Practice and Model Programs; Objective 1.3)

3. A health unit collects session feedback each week and changes the next session in response. Which reason to evaluate is this?

Accountability shows a funder or the public that resources had an effect. Weekly feedback used to adjust the next session serves the people running the program, not an outside funder.
Collecting feedback while the program runs and changing the next session in response is learning and improvement, which looks inward and is most useful while the program is under way.
The theory reason tests a predicted linkage, such as whether self-efficacy led to behaviour change. Session feedback used to adjust delivery is not a test of a theory.
Efficiency asks what it cost to reach the goals. Nothing in weekly session feedback concerns cost per result.
Correct answer: learning and improvement. The information needed is the four reasons in 2.1. Collecting feedback while a program runs and changing the next session in response is learning and improvement. Accountability is the tempting distractor because a health unit is a public body, but accountability means showing a funder or the public that resources had an effect; theory tests a predicted link, and efficiency concerns cost. (Section 1, 2.1 Four Reasons to Evaluate; Objective 2.1)

4. A funder requires grantees to use a model program or justify why none fits. What does "model program" mean?

Model programs are those with good evidence of effectiveness, and registries rate that evidence more highly when the program has worked in several places with different populations.
Who designed a program does not define a model program. The term refers to the strength of the evidence that the program works.
A model program can rest on any theory. The term is about evidence of effectiveness, not about which theory the program uses.
Longevity is not evidence. A program can run for years without evaluation; a model program is one with good evidence of effectiveness.
Correct answer: a program with good evidence of effectiveness, especially across settings and populations. The information needed is 1.3. Model programs are those with good evidence of effectiveness, and registries rate that evidence more highly when the program has worked in several places with different populations. A program designed by a government agency is tempting because funders often point to model programs, but who designed a program, which theory it uses, and how long it has run do not define the term. (Section 1, 1.3 Evidence-Based Practice and Model Programs; Objective 1.3)

5. A replication of a mental health curriculum shows no effect, but process data show teachers delivered only two of six sessions. What conclusion does the idea of fidelity support?

This is the superintendent's error in Worked Example 11.2. With only two of six sessions delivered, the result says little about the curriculum and a great deal about implementation.
A program not delivered as designed never put its theory to the test, so the result cannot disprove the theory behind the curriculum.
For a replication to count as an additional test of a model, all components must be delivered as designed. Two of six sessions is a low dose and a failure of fidelity, so it tests neither the curriculum nor its theory.
Nothing in the case concerns the baseline. The process data identify the explanation: the curriculum was not delivered with fidelity.
Correct answer: the replication lacked fidelity, so it is not a test of the curriculum or its theory. The information needed is fidelity and dosage in 3.2 and Worked Example 11.2. For a replication to count as an additional test of a model, all its components must be delivered as designed; two of six sessions is a low dose and a failure of fidelity. Concluding that the curriculum does not work in this population is the superintendent's error in Worked Example 11.2. (Section 2, 3.2 Process Evaluation: Did We Do What We Planned? and Worked Example 11.2; Objective 3.2)

6. Which of the following is an example of the kind of data an outcome evaluation collects?

Cancer morbidity and mortality after ten years measure the health problem itself, which makes them impact data, not outcome data.
A count of coalition partners is a record of what the program did, which is process data.
Random assignment records belong to a classic experimental design. They document how groups were formed, not a short-term effect of the program.
Outcome evaluation measures short-term effects such as knowledge change, using pre- and posttest, or baseline and follow-up, data on what the program expected to change.
Correct answer: pre- and posttest data on knowledge change. The information needed is 3.3 and Table 11.2. Outcome evaluation measures short-term effects such as knowledge change, health service use, and policy change, from baseline to follow-up. Cancer morbidity and mortality after ten years is tempting because it is also a measured effect, but it is the health problem itself, which makes it impact data; counts of coalition partners are process data. (Section 2, 3.3 Outcome and Impact Evaluation: Measuring Change Over Time, Table 11.2; Objective 3.3)

7. Why can long-term impact be measured only if the project is in place for more than three or four years or you can follow a cohort of participants?

Impact evaluation asks whether the health problem changed, and morbidity and mortality change slowly, so follow-up must run for years within a long program or by following participants as a cohort.
The lesson gives no funder rule against shorter evaluations. The limit comes from how slowly the health problem itself changes.
Short programs can and should collect baseline data. The issue is how long the follow-up must run before changes in disease appear.
Process evaluation runs throughout implementation and need not be finished first. The time frame for impact comes from how slowly disease rates change.
Correct answer: because changes in morbidity and mortality take years to appear and must be followed over that time. The information needed is 3.3.2. Impact evaluation asks whether the program affected the health problem that was its ultimate target, and changes in disease emerge slowly, so follow-up data must be collected for years, within a long-running program or by following participants as a cohort. The baseline option is wrong because short programs can and should collect baseline data; the issue is how long the follow-up must run. (Section 2, 3.3 Outcome and Impact Evaluation: Measuring Change Over Time; Objective 3.3)

8. In a logic model, which component is described as the criteria and tools for measuring outcome and impact, such as health data, survey or qualitative data, and measures of skill?

Inputs are the resources, staff, components, and funds invested, what the program has or uses, not the tools for measuring results.
Outputs are also counted, but they are the activities the program delivers, such as materials, events, and trainings, not the criteria for measuring outcome and impact.
Indicators and measures are the criteria and tools for measuring outcome and impact, such as health data, survey or qualitative data, and measures of skill. They make each column measurable.
The health problem comes from the epidemiological assessment and states what the program addresses. It is not a set of measurement tools.
Correct answer: indicators and measures. The information needed is the list of logic model components in 4.2. Indicators and measures are the criteria and tools for measuring outcome and impact: health data, survey or qualitative data, and measures of skill. Outputs is tempting because outputs are also counted, but outputs are the activities delivered, inputs are the resources invested, and the health problem comes from the epidemiological assessment. (Section 3, 4.2 The Parts of a Logic Model; Objective 4.2)

9. In the Harfield County example, why did the planners choose a social-cognitive approach with social network components?

This reverses the finding: Harfield County's newspaper rarely covered health, which is why the media were not chosen as a channel.
No trial had been run in Harfield County. The approach was chosen from the assessment of the county's strengths and weaknesses.
The assessment found large supermarkets were scarce, which made healthy food hard to reach. It was a weakness, not the reason for the approach.
The county's strength was organized groups with trusted leaders and movers. Leaders could model healthier diets, Social Cognitive Theory's observational learning, and the organizations could carry classes and cooking groups, a social network strategy.
Correct answer: the county's strength was its churches, service organizations, and veterans group, with leaders and "movers." The information needed is Worked Example 11.4. The assessment found few large supermarkets, a newspaper that rarely covered health, and organizations full of established leaders, so leaders could model diet change (Social Cognitive Theory) and the organizations could carry classes and cooking groups (a social network strategy). The newspaper option reverses the finding: the newspaper rarely covered health, which is why the media were not the channel. (Section 3, 4.2 The Parts of a Logic Model and Worked Example 11.4; Objective 4.2)

10. An evaluator randomly assigns every other person on a waiting list to receive a program now, with the rest as controls, and collects pre- and posttest data from both groups. Which design is this, and which problem does it raise?

A quasi-experimental design compares a similar community without random assignment. Here individuals on one list were randomly assigned, which is a classic experiment.
Benchmarking compares program results with outside data, such as national figures. This design forms its own control group by random assignment.
Randomly assigning individuals, such as every other name on a list, with pre- and posttest data from both groups is the classic experiment. Its problems are accidental exposure of controls and the ethics of withholding.
A periodic inventory tracks progress from a baseline with no comparison group. This design has both baseline data and a randomly assigned control group.
Correct answer: classic experimental design, with accidental exposure of controls and the ethics of withholding. The information needed is the ladder of designs in 5.1. Random assignment of individuals to intervention or control, with pre- and posttest data from both, is the classic experimental design, and selecting every other name on a list is a standard way of doing it. Quasi-experimental design is tempting because it also has a comparison group, but that group is a similar community, not one formed by random assignment. (Section 3, 5.1 Evaluation Designs: Making a Case; Objective 5.1)

11. A campus program recruits the heaviest drinkers. Their drinking falls at follow-up. Which confound most directly threatens the claim that the program caused the drop?

Participants recruited for extreme behaviour tend to move toward the average at follow-up whether or not the program worked. The heaviest drinkers are the lesson's own example of regression to the mean.
Diffusion of treatment is a control group receiving some of the intervention. The case describes who was recruited, not contamination of a comparison group.
Testing is change from familiarity with the survey. The threat here comes from where participants started, chosen for extreme drinking, not from answering the questions twice.
History is an unconnected outside event during the program. Nothing like that is described; the threat comes from recruiting the most extreme drinkers.
Correct answer: regression to the mean. The information needed is Table 11.4 in 5.2. Participants recruited because their behaviour is extreme tend to move toward the average at follow-up whether or not the program worked; with very high-risk behaviour, scores may improve only because they cannot get worse. Testing is the most tempting alternative because participants answer the same questions twice, but testing describes change from familiarity with the survey, not from where participants started; diffusion and history describe a contaminated control group and an outside event. (Section 3, 5.2 Confounds: What Weakens the Case, Table 11.4; Objective 5.2)

12. Which statement best describes the value of results that fall short of a classic experimental design?

The lesson states that results short of a classic experiment are not without merit, because programs run in the real world and success has many levels.
Outcome results short of an experiment also have merit. When time is short or barriers are many, an outcome evaluation may be the best that can be done.
Programs run in the real world and success has many levels, so there is no cookie-cutter standard. The evaluation should fit the program and the evidence needed, and its claim should match its design.
A control group added afterward cannot supply the baseline comparison or random assignment an experiment needs. The lesson's advice is to fit the design to the program and limit the claim.
Correct answer: they are not without merit; the evaluation should fit the program and the evidence needed. The information needed is 5.3. Programs run in the real world and success has many levels: sometimes getting a program operating is the achievement and a process evaluation is enough, and sometimes an outcome evaluation is the best that can be done. The option limiting such results to process evaluation is tempting, but the lesson says outcome results short of an experiment also have merit. (Section 3, 5.3 No Cookie-Cutter Standard; Objective 5.3)

13. What does theory contribute to a program that makes it evaluable?

Theory makes a program coherent and evaluable, but nothing guarantees funding. Funders expect evidence, which comes from evaluating what the theory predicted.
A theory-based program is a set of predictions. Each prediction names something that ought to change, so theory tells the evaluator what to measure.
Theory says what to measure; it does not rule out rival explanations. A comparison group is a design feature for addressing whether change would have happened anyway.
The budget and timeline are a separate part of a grant application. Theory appears in the plan, where it justifies why activities should change the factors.
Correct answer: theory tells you what ought to change, and therefore what to evaluate. The information needed is 2.2, with the same point repeated in 6.1. A theory-based program is a set of predictions, and each prediction names something an evaluation can measure. Theory also lends coherence to a program, but it does not guarantee funding, replace a comparison group, or supply the budget, which is a separate part of a grant application. (Section 1, 2.2 How Evaluation Relates to Theory; Objective 2.2)

14. Which parts does a grant application typically contain?

The six parts are the health problem, a plan justified by theory and analysis, an evaluation plan, staff and agency resources, organizational experience, and a budget and timeline.
A randomized protocol is one possible evaluation design, and registry ratings come from outside reviewers. Neither is a standard part of a grant application.
Position papers and media advocacy describe the work of advocacy organizations, not the parts of a grant application.
Funders may require a model program, but neither a list of model programs nor a letter from a program officer is one of the six standard parts.
Correct answer: the problem, a theory-justified plan, an evaluation plan, staff and resources, experience, and a budget and timeline. The information needed is the six parts in 7.2. Theory appears in the plan, which is a little like a PRECEDE-PROCEED analysis, and evaluation appears as its own part. The option listing model programs and a letter from a program officer is tempting because funders may require a model program, but neither is one of the six standard parts. (Section 4, 7.2 The Grant Application; Objective 7.2)

15. A graduate takes a position with the First Nations Health Authority helping communities define their own indicators of program success and control their own data. Which evaluation approach does this most resemble, and how does it differ from a classic experiment?

Benchmarking compares results with outside data such as national figures. The defining feature here is community control of goals, indicators, and data.
Empowerment evaluation is a collaborative effort in which the community identifies goals and how to measure them, on its own terms, in line with OCAP. It is not designed to isolate the program's effect.
A quasi-experimental design uses a comparison community to make a causal case. Helping communities define their own indicators is a participatory purpose, not a comparison design.
Historical record keeping tracks activities and participation for process evaluation. Defining indicators of success and controlling data goes well beyond tracking activities.
Correct answer: empowerment evaluation, in which the community defines goals and measures on its own terms. The information needed is 5.1 and the account of Indigenous health organizations in 6.4. Empowerment evaluation, drawing on Paulo Freire and participatory programming, is a collaborative effort in which a community identifies goals and the best ways to measure them, often instead of standard validated instruments, and it stands in contrast to experimental designs meant to produce rigorous comparative data. Benchmarking is tempting because community results may later be compared with outside data, but the defining feature here is community control of goals and data. (Section 3, 5.1 Evaluation Designs: Making a Case; Section 4, 6.4 Four People, and a Canadian Addition; Objectives 5.1 and 6.4)

16. In the newcomer families physical activity program described above, which pairing best matches the reasons to evaluate that matter most to each party?

This reverses the parties. The funder that pays asks for accountability and efficiency, while the delivering organization wants learning and improvement and a test of its theory.
Efficiency matters, but neither party's interest stops there. The health authority also needs accountability, and the organization wants feedback and a test of Social Cognitive Theory.
This reverses the roles. A funder asks for accountability; learning and improvement serves the organization running the program.
The funder needs to show that public money produced change against its goal and at what cost. The organization wants to adjust as it goes and to test whether role modelling raised self-efficacy and activity.
Correct answer: accountability and efficiency for the health authority; learning and improvement and theory for the organization. The information needed is the four reasons in 2.1. The funder needs to show that public money produced a change against the stated goal (accountability) and what each result cost (efficiency); the organization wants to adjust recruitment and sessions as it goes (learning and improvement) and to know whether role modelling and group support raised self-efficacy and activity as Social Cognitive Theory predicts (theory). The reversed pairing is the trap: the party that pays is the one that asks for accountability. (Section 1, 2.1 Four Reasons to Evaluate; Objective 2.1)

17. Which evaluation question about the newcomer families program could NOT be answered within its three years of funding?

Disease rates are the health problem itself, so this is impact evaluation, which needs a program in place for more than three or four years or a followed cohort.
Delivery and attendance are process questions answered by records kept throughout implementation, so they can be answered within three years.
Self-efficacy is a short-term outcome that baseline and follow-up surveys can capture within a three-year program.
Weekly minutes of activity measured from baseline to follow-up are a short-term outcome that can be captured within the funding period.
Correct answer: whether cardiovascular and metabolic disease fell. The information needed is the time frames in 3.3. Disease rates are the health problem itself, so this is an impact question, which needs a program in place for more than three or four years or a followed cohort. Delivery and attendance are process questions answerable during the program, and self-efficacy and activity minutes are short-term outcomes that baseline and follow-up surveys can capture within three years. (Section 2, 3.3 Outcome and Impact Evaluation: Measuring Change Over Time; Objective 3.3)

18. In a logic model for the newcomer families program, where do "weekly group walks, skill sessions, and family role-model events" belong, and which indicator fits that column?

Peer leaders and funding are inputs, the resources invested. The walks, sessions, and events are what those resources produce, which makes them outputs.
A pre- and post-program activity survey measures short-term outcomes, the effects of the activities. The walks and sessions are the activities themselves.
Walks, sessions, and events are activities the program delivers, so they are outputs. Process evaluation counts sessions delivered and dosage per family to test that column.
Long-term impacts are effects on health status measured over years. The activities that are meant to produce those effects belong in the outputs column.
Correct answer: outputs, measured by sessions delivered and dosage per family. The information needed is the logic model components in 4.2. Walks, sessions, and events are activities the program delivers, which makes them outputs, and process evaluation counts what was delivered and how much each family received. Inputs is tempting because peer leaders are needed to run the walks, but the leaders and the funding are the resources invested; the walks are what those resources produce. (Section 3, 4.2 The Parts of a Logic Model; Objective 4.2)

19. Which evaluation plan for the newcomer families program is realistic, and which confound must it address?

Withholding the program from half the families raises the ethical problem the lesson names, and no design leaves no confounds; experiments face accidental exposure of controls, among others.
Repeated measures from a baseline, benchmarked against provincial data, fit a three-year community program. Because families volunteer, those who join may be more motivated, which is selection bias.
Record keeping alone can show delivery but not change in activity, and regression to the mean concerns extreme starting values, not attendance sheets.
Benchmarking alone has no baseline for these families, and diffusion of treatment concerns a control group receiving the intervention, not provincial comparison data.
Correct answer: a periodic inventory with benchmarking, which must address selection bias. The information needed is the ladder of designs in 5.1, the confounds in 5.2, and the point in 5.3 that the evaluation should fit the program. Repeated measures from a baseline, compared with provincial data, are feasible for a three-year community program, and because families volunteer, those who join may already be more motivated, so improvement could reflect who joined rather than the program. The classic experiment is the trap twice over: withholding the program raises the ethical problem the lesson names, and no design leaves no confounds to address. (Section 3, 5.1 Evaluation Designs: Making a Case and 5.2 Confounds: What Weakens the Case; Objectives 5.1 and 5.2)

20. The newcomer families program's results are strong. Which division of roles would best turn them into evidence that could eventually reach a registry such as Health Evidence?

Program officers set success criteria and oversee without delivering, community organizations deliver with fidelity and keep process records, evaluators design measures, and academic partners add rigour and publish, which is how evidence reaches registries.
Program officers oversee but do not deliver, and funding announcements are written by the agency, not by the organization applying for funds.
Community organizations deliver programs, and registries rate evidence through outside review. A program cannot rate itself in a registry.
The evaluator designs measures rather than delivering the program, and publication is the work of academic partners, not of a program officer who oversees grants.
Correct answer: delivery and records by the organization, oversight by the program officer, measures by the evaluator, and publication by a university partner. The information needed is the funding process in 7.1 and academic settings in 6.3. Program officers set frameworks and success criteria and oversee without delivering, community organizations deliver and keep the records that show fidelity, and academic partners add rigour and share results through publication, which is how a program's evidence reaches the reviews and registries that make it usable elsewhere. The options that give the program officer delivery tasks repeat the most common confusion about that role. (Section 4, 6.3 Consulting, Industry, Health Care, and Academic Settings and 7.1 Two Ends of the Funding Process; Objectives 6.3 and 7.1)

✦ Pass every section knowledge check with 100% before submitting