basics
What is the difference between probability sampling and non-probability sampling?
Probability: each person has an equal, nonzero chance of being selected for the sample. (Random sampling, stratified sampling, cluster sampling)
Non-Probability Sampling: certain individuals have a higher chance of being selected than others. (snowball sampling, convenience sampling)
True or false: standardization makes effect sizes easier to interpret.
False - unstandardized actually maintains more consistency with the results.
What are person-related response biases? (3)
Social desirability, acquiescence, extremity: even with a very well-validated scale, it may not work the same way every time due to the participants’ responses.
What does the phrase, "don’t throw the baby out with the bathwater" mean?
Not all lab studies are poor quality just because they lack ecological validity.
What's a common cause?
A variable independently influences the independent and dependent variables, which causes a false correlation to appear.
Define open data, open source and open access.
Open access: anyone should be able to read the articles. Some journals are doing this faster than others. When journals require people to pay in order to publish findings, some people conduct poor research and just “pay their way” through.
Open data: allowing others to access your data to run replications. Many journals require this.
Open source: the tools we use (e.g. Jamovi) should be accessible to everyone.
What's the difference between unstandardized and standardized effect sizes?
Unstandardized ES (B): set in the measurement units of the outcome variable (e.g. “minutes more sleep”
Variables have meaning with reference to what is being measured
Make the most sense when measurement units of outcome variable are physical (minutes, pounds, liters, etc.), but a lot of psych units are ARBITRARY!
Range restriction NOT an issue–effect size is based on rise/run, which is unaffected by restricting the range (slope of regression line is constant)
Standardized ES (r, r-squared, cohen’s d, Beta, eta-squared)
Used when no meaningful measurement units are present (E.g. “perception of sleep quality”)
Cannot be compared across studies, since standardization depends on sample characteristics (relative group sizes,. Variance within the design, standard deviation) - more “dirty” than unstandardized effect sizes.
Watch for range restriction in standardized samples! As the SD becomes smaller, the z-scores become inflated, which will positively skew the standardized effect size.
What is the difference between an interrupted time-series and control group time-series study design?
Interrupted Time-Series Design: Similar to the time-series design, but more points are added between the pre- and post-test points to give us a better idea of the trends, which strengthens causal inference.
Control Series Design: Similar to the Interrupted time-series, but adding a control group that receives no treatment and is measured at the same points.
What are 2 ways to prevent threats to internal validity?
Random assignment, implementing a control group
Define the question-related response bias of category anchoring.
Someone sticking to only positive/negative/neutral answers
What's the difference between logical positivism and humanism?
Logical positivism: empiricism and logic are the sole bases of meaningful knowledge. Scientists’ personal beliefs have no effect on science.
Humanism: Scientists’ personal beliefs have influence over the theory, the methods, and the interpretation of the results.
Pick the correct statement:
1. The probability that the result is due to chance is less than .05
2. The probability that the null hypothesis is true is less than .05
3. The probability of the data, given the null, is less than .05
4. The probability that a type I error was just made in rejecting the null hypothesis is less than .05
3 is correct - when the null is true, the possibility is less than .05
What are the different response formats on surveys? (4)
Comparative Rating Scales: ranking items compared to each other. (E.g. place these in order from most-least desirable.)
Itemized rating scales: multiple-choice questions
Graphic rating scales: given a line, place your response on the line (E.g. Likert Scales)
Numerical rating scales: “On a scale of 1-100…”
What are the possible threats to internal validity?
History
Maturation
Testing effects
Instrumentation
Attrition
Statistical regression to the mean
Selection bias
Reactivity
Experimenter bias
What's a yoked control group?
Yoked Control Group: form of matching where a variable expected to be different across conditions is equalized. (either post-hoc or impromptu)
Each person → matched with someone who will have the same experience as a part of the study
DIFFERENT from matching → about the experience in the study, not about characteristics that the participants have that already exist.
Name 5 issues with the NHST
promotes dichotomous thinking and is sample driven
can't DIRECTLY test alternate hypothesis
replication fallacy
language factors (say "statistically significant" instead of "highly significant")
"nil hypothesis" (if effect is 0 in a population, leads to inaccuracy in p-values bc the data will look more extreme than it is)
people keep using it wrong
original thought was that null and alternate hypothesis are mutually exclusive and exhaustive, but this doesn't really apply to issues of probability (they're very definite statements)
What is the most significant testing crisis right now? (Hint: it's related to the p-value)
P-hacking: trying to change the p-value of a study so that it appears statistically significant.
What are the advantages and disadvantages of observational studies?
Advantages of observational studies:
Help maintain the context surrounding research (risk and protective factors, identifying individuals at risk)
Drawbacks of observational studies:
Unspecified constructs/operationalizations
Selecting groups (important to match)
Direction/Type of influence
What are the 3 primary components of external validity?
Structural: How a study is carried out.
Functional: are the psychological processes that operate in the study the same ones that operate in the natural setting? (pertains primarily to ecological validity)
Conceptual: are the questions that are being studied similar/corresponding to problems that are considered important in the natural setting? (pertains primarily to ecological validity)
Even in true assignment, we cannot assume that treatment is the sole differentiator between groups (nonspuriousness) without presence of ____ as the control, and ____.
experimental groups, random assignment of participants
Good theories have... (name 7)
Logical consistency
Falsifiability
Agreement with known data
Clarity
Parsimony (the simplest theory is often the most useful → Occam’s razor)
Consistency
Applicability to real world
When do SMALL effect sizes carry more meaning?
Minimal manipulation of IV resulted in strong outcome change.
Effect is there when it wasn’t expected.
Outcome variable is highly important (e.g. human life)
Effect is present on a notoriously hard-to-change outcome variable!
Small changes in health-related variables, when spread over the entire pop, are large!
What are the 5 single-case experimental study designs?
“A-B-A-B” a.k.a. “Reversal”: switching the IV “on” and “off” again.
“A-B” design: when changes from the IV are permanent
“A-B-C-B”: C is an additional control: removing something else, which acts as another baseline to test if C acts as a control.
Multiple baseline design: A staggered interruption of the IV, which allows us to see the impact of the onset of the IV while still controlling for history and maturation.
Simultaneous treatment design: tests the relative effectiveness of multiple treatments
Changing criterion design: When a gradual/slow change is expected, IV is not withdrawn, but instead the change is set across an increasingly difficult criterion variable.
What are the 3 primary sources of confounds?
Demographics
Treatment variable level changing might influence other variables we don’t know about
Measurement invariance (bias at the measurement level)—this is an issue with construct validity.
What are 4 ways to test for mediation?
B&K’s Causal Steps: test IV→DV and then Mediator →DV, and then put them altogether in regression: if the correlation coefficient for IV→DV decreases, you have a mediator.
Joint Significance: if the relationship between IV and mediator is significant, and then the relationship between the mediator and DV is significant, then the mediator is significant.
Sobel test: multiplying regression coefficients for IV→Mediator and Mediator→DV = significant? Then the mediator is significant (not recommended anymore)
Bootstrapping: idk man never understood it here’s a little face 🙂