Experimental Research Fundamentals
What is the key difference between experimental and non-experimental research?
Experimental research can identify and test causal relationships.
Whats the key difference between extraneous variables (i.e., extraneous variables, confounding variables) and other third variables (i.e., mediators, moderators)?
External validity is concerned with assessing the degree to which the results of a study generalize _______________ and ________________.
Across Settings
Across People
What are two things multi-factor designs allow that single-factor designs do not?
- Test unique effects of multiple IVs
- Test if IVs interact with each other to affect the DV
What is one key limitation of behavioral measures?
Behavioral measures can sometimes be problematic because people react unnaturally when they know they are being observed.
What are the 3 things you need to identify a causal relationship?
Evidence Covariation
Establish Temporal Precedence
Eliminate Extraneous Variables
Raya finds that receiving critical feedback improves employee performance on average — but this effect is much stronger among employees who report high psychological safety at work. Among employees who feel unsafe speaking up, critical feedback has no effect on performance and sometimes worsens it.
This is an example of which kind of 3rd variable effect?
Threats to validity can be categorized into what 3 buckets?
People are different.
People change.
People act differently under observation.
Dr. Rafael Mendez finds that messages attributed to a public health researcher generate more support for decriminalization overall than those attributed to a law enforcement official — but this advantage disappears when the message is framed around what society stands to lose by maintaining criminalization rather than what it stands to gainfrom decriminalization, because that framing disproportionately boosts the law enforcement source's persuasiveness while leaving the researcher's unaffected.
Identify the main and interactions effects in the above scenario.
Main Effect: Overall benefits of public health researcher driven messages.
Interaction Effect: Benefits of that change with loss / gain framing.
Identify the Conceptual IV & DV and the Operational IV & DV in this scenario: Marcus is studying whether perceived legitimacy of law enforcement shapes cooperation with police. He surveys residents in three neighborhoods that recently experienced different levels of publicized police misconduct, asking them how fair and trustworthy they believe local police to be and whether they would be willing to report a crime they witnessed.
Conceptual IV: Perceived Legitimacy
Conceptual DV: Cooperation with/ Policy
Operational IV: "How fair and trustworthy..."
Operational DV: "Willing to report a crime..."
What 3 strategies comprise the "gold standard" of experimental research?
Random Assignment
Control Group
Controlled Experimental Environment (i.e., eliminate all non-experimental variables)
Dr. Mary Chen finds that adolescents who experience frequent peer rejection perform worse academically. She notes that rejected students tend to miss more school, which independently drags down their grades. However, even among students with similar attendance records, peer rejection still predicts lower achievement — in those cases, ongoing conflict with classmates appears to be creating enough disruption in the classroom to interfere with learning.
Identify and label (i.e., which kind) all the 3rd variables in this scenario.
Attendance and Class Conflict are both mediators.
Which types of validity of applied researchers more likely to prioritize? Why?
External & Ecological
Their solutions need to be applicable to a variety of different types of people and must be robust to the extraneous variables that exist in the real world for them to be adopted.
Name and describe 1 advantage and 2 drawbacks of within-subjects designs - as well as one solution that can help prevent against them.
Benefits: Reduce impact of individual differences & Requires fewer resources.
Drawbacks:
Fatigue Effects: When you see a decrease in performance as a result of repeated exposure to the independent and dependent variable
Practice Effects: When you see an improvement in performance as a result of repeated exposure to the dependent variable
Contrast Effects: When the response to the second condition is affected because the participant contrasted the conditions to each other
SOLUTION: Counterbalancing
Tomas is surveying employees about workplace culture. He randomly selects equal numbers of participants across all three levels of the organization — frontline staff, middle management, and senior leadership — to participate in his study. Despite multiple reminders, only 10% of the hourly workers in his sample have completed the survey, likely because they have less discretionary time and are more skeptical about confidentiality. His result show very high levels of satisfaction with the organizational culture.
Answer the following regarding this scenario: (1) What sampling strategy is Tomas using; (2) What type of bias may be impacting his results?; (3) What strategy could use to correct for that bias?
Stratified Random Sampling
Non-Response Bias
Incentives
Jose notices that in neighborhoods where community policing programs were introduced, crime rates dropped over the following two years. He concludes that community policing reduces crime. However, when he examines the data more closely, he finds that crime rates dropped at nearly identical rates in comparable neighborhoods that received no such program.
Which of the 3 things you need to identify a causal relationship are missing here?
Failed to evidence covariation - given that the trend is the same across communities where the program was introduced and where it wasn't.
Failed to eliminate extraneous variables.
Emory is testing whether a memory supplement improves recall, and whether its effectiveness depends on a person's baseline memory ability. Neither participants nor the researchers administering the pills know who receives the active supplement versus a placebo pill. He finds that the supplement increased memory in the experimental group. However, post-study interviews reveal that the majority of participants in the experimental condition were able to guess the pill they received because of its taste and unusual markings.
Answer the following questions regarding the above experiment: (1) Identify any potential mediators or moderators explored in this study; (2) Identify any extraneous of confounding variables in this study; (3) Identify any strategies used to mitigate extraneous variables in this study.
Moderator - baseline memory
Extraneous - identification of pill by experimental condition.
Strategies - double-blind.
Diane studies whether exposure to a sympathetically framed news story about a formerly incarcerated person reduces support for mandatory minimum sentencing. She recruits undergraduate students through the psychology subject pool at the John Jay College of Criminal Justice, brings them into the lab to read the article in a quiet lab cubicle, and then enters the room when they are done to measures their policy attitudes via a researcher-administered questionnaire.
Answer the following questions based on the above scenario: (1) Identify two types of experimental validity that are at risk in this design. (2) For each one, identify one way you could fix it.
Ecological - lab environment not how people engage with political / policy process. (Fix: online administration)
External - students at a school specializing in criminal justice may not generalize to non-college students or even students at different schools. (Fix: representative sample, multiple studies with different samples)
Internal - presence of researcher during the questionnaire process can lead to heighten social deniability effects while also introducing possibility of demand characteristics or reactance effects. (Fix: anonymous survey follow-up at a later time)
Jasmine Okafor is examining how academic self-confidence, course format, instructor feedback style, and assignment difficulty interact to shape student motivation. She recruits students who report either high or low academic self-confidence - with half coming from her online courses and half from her in-person courses. She then randomly assigns them to receive either positive, neutral, or constructive critical feedback on a practice assignment before giving them a follow-up task that is either very easy or genuinely challenging. Once the follow-up task is complete, she has them answer a series of questions about their internal motivation for academic success.
Answer the following as it relates to the above scenario: (1) What is the factorial nomenclature for this design? (2) How many main effects are being tested and what are they?
4 Main Effects - each of the IVs on academic motivation.
Dr. Harmon is studying financial stress among college student's mental health. To measure financial stress, he asks participants a single question — "How stressed are you about paying your tuition?" — rated on a scale from 1 (not at all) to 7 (extremely).
Answer the following as it relates to Dr. Harmon's measure: (1) What type of measure is it?; (2) What level of measure is it?; (3) Identify and explain the core validity issue with the measure?
Type: Self-Report
Level: Ordinal
Validity Issue: Content Validity - financial stress is broader than paying for tuition.
A city health department reports that neighborhoods with more fast food restaurant openings over the last 5 years have higher rates of obesity. Officials conclude that eating fast food is causing residents to gain weight and propose limiting new fast food permits as a public health intervention.
Answer the following questions based on the above scenario: (1) Which of the 3 things you need to identify a causal relationship are missing here?; (2) How would you design/ redesign the study to fix these issues?
All 3!
Dr. Fatima Al-Rashid is studying whether a structured sleep hygiene program reduces chronic pain, whether this effect is explained by improvements in mood, and whether the program works better for patients with higher baseline pain levels than those with lower ones. She exposes all participants to the full program but varies the order of its two components across time points, collecting mood and pain measures throughout. She finds that the program reduces pain, that mood improvements partially account for this effect, and that patients with higher baseline pain show the greatest gains. However, exit interviews reveal that participants who completed the more intensive component first reported feeling significantly more fatigued during the second phase, which may have independently affected both their mood and pain ratings.
Answer the following questions regarding the above experiment: (1) Identify any potential mediators and moderators explored in this study; (2) Identify any extraneous and confounding variables in this study; (3) Identify any strategies used to mitigate extraneous variables in this study.
Nadia is studying whether wearing activity tracking devices increases physical activity. She recruits participants through flyers posted at a gym and gives everyone the same wearable device — telling one group it will monitor and report their exercise and the other that it will only monitor and report their blood sugar. At the end of the study, participants filled out a survey saying how many minutes they worked out during the study period. Self-reported minutes are nearly identical across groups, but the device data tells a different story: the group who believed their exercise was being monitored logged significantly more active minutes.
Answer the following questions based on the above scenario: (1) What 3 threats to validity are being illustrated in this scenario? (2) Which types of validity are these threats most likely to impact? (3) How can you change the study to address those validity concerns?
Selection Bias - recruitment in gym is likely to draw a certain kind of person who is already interested in fitness / has higher baseline fitness activity than general population. External Validity.
Hawthorne Effect - participants who are told their activity is being tracked could have performed better because they knew they were being observed. Internal Validity.
Social Desirability - Self-report deviation from the activity trackers suggests people in non-experimental condition were responding in a way that would reflect favorably on them. Internal Validity.
Dr. Amara Osei is studying how mood and time of day interact to influence creative thinking. She recruits 40 participants and brings them into the lab on two separate days — once after a positive mood induction and once after a neutral mood induction. On each visit, participants complete a creative problem-solving task at four different points: 8am, 12pm, 4pm, and 8pm. Every participant completes all eight conditions.
Answer the following regarding the above scenario: (1) Is this a within or between-subjects design? (2) What is the factorial nomenclature for this design? (3) What risk/drawback is inherent in this design? (4) How could the researcher protect against that risk?
Within-Subjects
2 (mood Induction) X 4 (Time of Day)
Risk: Practice Effects / Fatigue Effects
Solution: Changing the specific task. (Counterbalancing won't really help here)
Dr. Chinwe Obi designs a 10-item math anxiety scale for middle schoolers, administering it to the same cohort twice over one month with no major assessments in between. She runs statistical analyses to check whether all 10 items correlate with each other, then validates scores against performance on a standardized math test, expecting higher anxiety to predict lower scores. She also correlates her measure with measures of math self-confidence, GPA, and engagement with student groups on campus. When she's done, she conducts focus groups with students to ask them if looks like a good measure of math anxiety to them, prompting them to think of any items they feel may be missing from the measure.
Identify all the validity and reliability strategies Dr. Chinwe Obi is using to validate her measure of math anxiety.
Internal Reliability
Test-Retest Reliability
Criterion Validity
Discriminant Validity
Face Validity
Content Validity