Basic
The 3 types of Evaluation
Formative, Process, and Summative
An Example of History Bias
Exposure to previous intervention, political or social event
The “counterfactual"
The unobserved outcome is called the “counterfactual” outcome
What would have happened to the intervention group had the intervention not be given to the group
The "Magic" Formula for Outcome objectives
Specific
Measurable
Attainable
Realistic
Timebound
Recruitment, Refusal, Retention, Attrition
List strategies to keep stakeholders involved
Keep a communication line
Clearly defined tasks and timeline
Engage them for tasks they are qualified for
Incentive when stakeholders are not professionals
Two examples of measurement bias
Social Desirability, Recall Bias, Interviewer Bias
A reflexive control design
one group compared against itself, weak design for internal validity, pretest serves as counterfactual
Characteristics of a Health Goal
Defines the health change, Indicates a direction (increase or decrease), Specifies target group, Specifies location
2 Common Problems with Survey Questions
too long
complex words
leading questions
inappropriate reference frame
double barreled
using absolutes
Characteristics of Internal Evaluators
more knowledgeable about program and context
less objectivity
more community trust
cheaper
Preventing Selection Bias
Control group
Random assignment
Large sample
Ensure a complete sampling frame
The 3 Steps of Matching
1. Identify Targets for Intervention
2. Select relevant criteria
3. Build a control group
Give an example of an external factor (Logic Model)
The unexpected loss of funding
A pandemic like Covid-19
An indicator of the dose delivered (numerator and denominator)
% of training sessions provided over the total number of training expected to be delivered
% demonstration exercises implemented over the total number of exercises expected to be delivered
% flyers/handouts distributed
What is the purpose of the Evaluability Assessment?
Determine if the program is ready for a summative evaluation
Prepare the program to generate all the necessary conditions to be evaluated
An example of maturation bias
Teens aging, and becoming better educated
When is RCT design not appropriate
Sample too small, random assignment unethical, too costly or time consuming
Rationale for conducting a formative evaluation
Evaluate the nature of the problem
Help determine the target population
Ensure intervention is culturally appropriate
Pre-test instruments and materials
Determine the measurement procedure
There are three Groups of Matching Variables
1. Composition
2. Predisposition
3. Experience
Reasons to Include Stakeholders
increase buy in, foster trust and facilitate evaluation process, foster strategic thinking, impact quality/use of findings, increase evaluation capacity
Threats to Validity Related to the Program Implementation
Altering TOC, altering quantity or quality of activities, inappropriate target audience, etc
Conditions for an RCT
There is some uncertainty about the value of the "experimental" intervention
There are sufficient resources (financial, human)
The design meets ethical standards
The evaluation design was planned at the start of the program
There is a control group, and participants have been assigned to the groups using randomization
How do we know that we are using the right theory?
Review the literature
Use evidence from scientific studies
Ask the experts
Stratified Sampling
The population is separated into subgroups to obtain samples.