Threats to Validity
100: How would you define "selection bias"?
When groups differ significantly from target population - study results can be biased due to confounding
100: Describe what happens when you UNDERESTIMATE the effect size for a study.
You get a larger sample size - leads to Type I error.
100: Describe the difference between theory of cause and effect and theory of implementation.
Theory of cause and effect: shows how the independent variable is connected to the mediating and dependent variables.
Theory of implementation: shows the what, where, how, when, and who of a program (ex: timeframe it happens, who's implementing it, what resources were needed for the program, etc.)
100: Describe the difference between validity and reliability.
Validity: you're measuring what you're intending to measure
Reliability: you're getting consistent results each time you measure something
200: What's the difference between "maturation" and "history" threat?
Maturation: occurs naturally over time, examples include aging, development, and role transitions.
History: when external events occur that can affect the outcome of the intervention.
200: Describe the difference between a Type I error and a Type II error in a programmatic sense, and explain what happens with each error in terms of resources, time spent, recruiting participants, etc.
Type I error - saying a program works when it doesn't, etc.
Type II error - saying a program doesn't work when it does, etc.
200 (process): Describe the importance of a process evaluation (i.e. why do we do it?)
200: What type of validity does the response rate of a survey typically effect?
External validity
300: describe the difference between random sampling and random allocation, and specifically note what threats to validity these study techniques control.
Random sampling controls for external validity threats by randomly sampling participants in a target population - this sample will be more representative of the population.
Random allocation controls for internal validity threats through the researchers assigning participants to either the intervention/exposure group or the control group.
300: What happens when you OVERESTIMATE the effect size in a power calculation?
300 (formative): You're on a team of educators who are developing a statewide program to help underrepresented high school students attend college courses while in high school at no cost to them. What's one formative evaluation question you could ask?
Anything along the lines of finding potential flaws prior to implementation, figuring out if the staff developing the program have the same goals in mind, figuring out potential roadblocks to data collection during implementation, etc.
300: Describe construct validity.
Measure getting at the latent concept (”true” value)
◦ Hard if there's no true gold standard exists (focus often on
criterion validity to start to prove construct)
◦ The ultimate “validity” conclusion
400: What happens if a threat to validity drives AWAY from the null hypothesis, and the study yields a significant difference between the intervention group and control group?
It remains a threat to validity - there may be confounding here.
400: Name at least two bivariate analysis methods for proportions.
Chi-square,
Fisher's exact, McNemar's test, simple relative measures OR, RR, PR or absolute measures with CI
400 (outcome): The goal of RE-AIM is to encourage program planners, evaluators, readers of journal articles, funders, and policy-makers to pay more attention to essential program elements including what kind(s) of validity?
External AND internal
400: describe Alternative Form Reliability and how you would assess for this
Looking at whether a measure can be given in a different form or medium. You would administer different forms of the same measure to the same
individuals at different points in time
◦ Ex. SAT: computer v. paper-and-pencil
◦ Ex. Long v. short form
Then calculate correlation between the scores on the two forms
If the scores correlate highly, then the scales are reliable
(Standard: r ≥ 0.61)
500: An evaluation is conducted of an educational intervention program for low-income children with asthma. Thirty buildings in public housing throughout Boston are selected and matched on demographics into 15 pairs. Each building in each pair is subsequently randomized to intervention or control status. This results in 15 intervention buildings and 15 control buildings. Pre- and posttest measures assess mean asthma severity in the intervention and control buildings.
What threats to validity are the researchers controlling for with this study design?
Contamination and selection
500: Describe when you might use a multivariable linear regression for analysis, and describe what it would show.
When you have multiple potential confounders that you want to account for.
Shows the relative contribution of your program (exposure) variable—given the presence of the other adjustment variables in the equation—to variation in the dependent variable
500 (outcome): A team of evaluators examined how effective a program geared toward supporting first-generation students in their first year of college was in retaining its students. The evaluators found that retention rates among this population of students increased by 24%, and want to consider different factors that led to this increase. How might they go about doing this?
Any questions regarding the outcome: To what extent can the outcome be attributed to the program, what was the influence of other factors, etc.
500: describe Criterion Validity and how you would assess for this.
Correlates with other criteria as we would expect
◦ Predictive: Ability to predict something we expect it would predict
◦ Concurrent: Performs similar to another measure at the same time
point that should measure the same concept
• Example:
→Self-report of an illness correlates with a biometric test
→New self-esteem scale correlates with other self-esteem scale
◦ Discriminant: Differentiate from concepts unrelated