Open Science & Research Integrity
Define open science and name three specific practices that fall under it.
Open science means making the whole research process transparent and freely available (data, materials, methods, results) so others can check and build on it. Practices: preregistration, registered reports, open data, open materials/code, open-access publishing, and replication studies.
Name and briefly describe the three fundamental features of science.
Systematic (careful, structured observation), empirical questions (questions answerable by observation), and concensus or verification (findings are published so others can scrutinize and replicate them)
Name three sources of research questions, and explain why a literature review matters even when your idea came from everyday observation.
Sources: informal observation, practical problems, past research (gaps, contradictions, “future directions” sections), and theory. A literature review shows whether the question has already been answered, helps you refine it, and points you to existing measures and designs.
What is a construct, and why must it be operationally defined before a study begins?
A construct is an abstract psychological attribute that can’t be observed directly (self-esteem, loneliness). Operational definitions turn it into something measurable, so the study can be conducted consistently, replicated by others, and clearly communicated.
Define reliability and validity, and explain why a measure can be reliable without being valid but not valid without being reliable.
Reliability is consistency; validity is whether the measure captures what it claims to. A scale that always reads 5 lbs heavy is reliable but not valid. A measure that gives random, inconsistent results can’t be accurately measuring anything, so reliability is necessary but not sufficient for validity.
Name the two features that make a study a true experiment and explain why they allow causal conclusions.
Manipulation of the IV and random assignment to conditions. Manipulation establishes that the cause comes first. Random assignment makes the groups equivalent at the start, which rules out pre-existing differences (third variables) as explanations.
Name and define the three principles of the Belmont Report.
Respect for persons: autonomy, informed consent, and protection for those with diminished autonomy.
Beneficence: maximize benefits and minimize harm.
Justice: distribute the burdens and benefits of research fairly when selecting participants.
Explain the purpose of inferential statistics, using the terms sample and population.
Researchers can rarely study a whole population, so they study a sample. Inferential statistics let them judge whether patterns in the sample are likely to hold in the population or could plausibly be due to chance (sampling error).
What is the “file drawer problem,” how does it distort the published literature, and which open science practice most directly fixes it?
Studies with non-significant results go unpublished (“in the file drawer”). The literature then over-represents positive findings and overestimates effect sizes.
Registered reports fix it most directly because journals accept the paper before results are known.
Public preregistration registries and publishing null results also help.
Your grandmother insists “full moons make people act crazy” because “everyone has always known that.” Name this method of knowing and explain one specific way it fails.
Authority(grandmother) / Consensus (everyone knows, repeated often)
It fails because repetition isn’t evidence. It also resists falsification and invites confirmation bias, since people remember the chaotic full-moon nights and forget the calm ones.
Name and explain three criteria scientists use to judge whether a theory is a good one.
Any three: falsifiability (it could be proven wrong); parsimony (explains a lot with few assumptions/ simplest explanation in alignment with evidence); breadth (accounts for results from many hypotheses; predictions (generates new testable hypotheses); empirical support (consistent with data).
Name three general approaches to operationalizing a construct and give one example of each for “loneliness.”
Self-report (scores on the UCLA Loneliness Scale); behavioral (number of social interactions logged per day, or time spent alone); physiological (e.g., stress hormone levels). Any reasonable example of each approach counts.
Name the three main types of reliability and describe the data you would collect to assess each.
Test-retest: give the same measure to the same people twice and correlate the scores. Internal consistency: check whether the items correlate with each other (split-half or Cronbach’s α). Inter-rater: have two or more observers code the same behavior independently and correlate or compare their ratings.
Define internal and external validity and explain why increasing one often decreases the other.
Internal validity is confidence that X caused Y. External validity is whether findings generalize to other people, settings and times. Tight lab control and standardized procedures boost internal validity but make the situation artificial, which reduces how well the results generalize to the real world.
Describe one historical case of unethical research OTHER than Tuskegee and identify the ethical principle(s) it violated.
So many!
Define a Type II error, give an example from a psychology study, and name two things that make Type II errors more likely.
A Type II error is failing to reject the null hypothesis when it is actually false, i.e., missing a real effect.
Example: a therapy really reduces anxiety, but the study concludes it has no effect. More likely with small samples (low power), small effect sizes, unreliable measures, or a very strict alpha level.
Explain how preregistration protects against BOTH p-hacking and HARKing. Give one separate mechanism for each
P-hacking: it locks in the analysis plan ahead of time (DVs, exclusion rules, covariates, sample size), so the researcher can’t shop around for a significant analysis.
HARKing: it timestamps the hypotheses, so a post-hoc finding can’t be presented as predicted.
Exploratory analyses are still allowed but must be labeled exploratory.
A headline reads: “Study PROVES video games cause aggression.” Rewrite it the way a scientist would and give two reasons for the change.
You are all scientists now. DO NOT use the word prove in your APA reports.“Study finds evidence that violent video game play is linked to increased aggression.” Reasons: (1) conclusions rest on probability, since inferential statistics always carry some chance of error; (2) scientific knowledge is tentative and can be revised or overturned by new evidence. A single study also can’t settle a question without replication.
Explain how theories generate hypotheses and how study results feed back into the theory. What should happen to a theory when a well-designed study fails to support its prediction?
A hypothesis is deduced from the theory (“if the theory is true, then X should happen”), then tested. Support increases confidence but doesn’t prove the theory. Non-support suggests the theory may need revising, needs boundary conditions, or should eventually be abandoned. Researchers first rule out methodological problems and replicate before rejecting it.
Classify each by level of measurement and justify your answer:
(a) a runner’s bib number
(b) finishing place (1st, 2nd, 3rd)
(c) race-day temperature in °F
(d) minutes to finish the race.
(a) Nominal: the numbers are only labels.
(b) Ordinal: order is meaningful but the gaps between places aren’t equal.
(c) Interval: equal intervals but no true zero (0°F isn’t “no temperature”).
(d) Ratio: equal intervals and a true zero, so 60 minutes really is twice as long as 30.
Distinguish face validity from content validity, and explain why face validity alone is weak evidence.
Face validity is whether the measure appears on the surface to measure the construct. Content validity is whether it covers all facets of the construct as defined, usually judged by experts. Face validity is subjective and intuitive, and intuitions can be wrong.
Choose the best design (experimental, correlational or observational) for each question and justify your choice:
(a) Does a new reading app improve 3rd graders’ vocabulary?
(b) Is childhood trauma related to adult anxiety?
(c) How often do toddlers share toys during free play?
Experimental - manipulate app usage
Correlational - assess relationship unethical to manipulate trauma
Naturalistic or Laboratory observation
Explain how the history of unethical research has lasting consequences today, both for affected communities and for the quality of science.
Communities: deep mistrust of medicine and research, which contributes to lower clinical-trial participation and to health disparities.
Science: underrepresented groups make samples less diverse, which hurts external validity, so treatments may not be tested on the people who need them
In R, you wrote code to create an object called hours holding the values 6, 7, 5, 8, 9. You the type Mean(hours). The following error appears "Error: could not find function “Mean”. Explain what you think happened and how you would troubleshoot stating two specific things you would do.
Help: ?mean or help(mean)
Using tab to find the function
Using an LLM or stackflow search
Course materials
Instructor/TA (ideally exhausting other options first)
A researcher runs 20 participants per group and checks the p-value. If it’s not significant, she adds 10 more per group and checks again, and stops as soon as p < .05. Name this practice and explain why it’s a problem even though she never fakes data.
It’s p-hacking through optional stopping (data peeking). Every check is another chance to get a false positive, so the real Type I error rate climbs well above .05. The fix is to set the sample size in advance with a power analysis, or preregister a stopping rule.
A company claims its “NeuroGlow” supplement boosts memory. When customers report no improvement, the company says they “didn’t take it with enough positive intention.”
Name two features of pseudoscience on display and describe what a scientific test of the claim would require.
Features: the claim is unfalsifiable, with excuses invented after failures to explain them away; it relies on testimonials instead of data; and it lacks peer review; positive intention is not defined
A scientific test would be a randomized, double-blind, placebo-controlled trial with an operationally defined memory outcome, specified in advance and published so others can replicate it.
Turn this vague claim into a testable hypothesis using a correlational design “Social media is bad for teenagers.” Your hypothesis must state the predictor, outcome, the predicted direction and the population.
Example: “Adolescents aged 13–17 who use social media 3+ hours per day will report higher depressive symptoms (CES-D scores --some depression survey) than adolescents who use it less than 1 hour per day.” A full answer is specific, measurable and falsifiable, and names a direction.
Students choose whether to study with music or in silence, and later exam scores are compared. Identify the IV and DV, classify each as quantitative or categorical, and name a plausible confound and how it creates an alternative explanation.
IV: study condition (music vs. silence), categorical. DV: exam score, quantitative. Possible confounds: prior GPA, personality, attention difficulties, or study location. For example, students who pick silence might be more conscientious, and conscientiousness, not silence, could explain their higher scores.
You’re studying procrastination. Explain the factors you would weigh in deciding whether to use an existing measure or create your own, and which option is usually preferred.
Existing measures are usually preferred because they already have reliability and validity evidence, save time, and make results comparable with past research. Creating your own is justified only when no measure exists, existing ones don’t fit your construct definition or population, or they’re outdated. A new measure then needs pilot testing and its own reliability and validity evidence.
A college invites students who scored in the top 5% on a September stress screening into a mindfulness program.
In November their stress scores have dropped. There is no control group.
Name at least three internal validity threats and explain how each could produce the drop without the program working.
regression to the mean
maturation
history
testing
attrition
Classify each as active deception, passive deception or no deception:
(a) Participants get false feedback saying they scored low on an intelligence test.
(b) Participants aren’t told that their gaze at food images is being recorded.
(c) A “fellow participant” who is actually a research assistant argues with the real participant.
(b) Passive
(c) Active
Debrief after!!
Name the correct correlation coefficient for each pair: (a) hours of TV per week and anxiety score
(b) class rank and race finishing place
(c) pet ownership (yes/no) and loneliness score
(d) smoker (yes/no) and lung disease diagnosis (yes/no).
Then say which one is mathematically equivalent to an independent-samples t-test and why.
(a) Pearson’s r
(b) Spearman’s rho (both ordinal)
(c) Point-biserial
(d) Phi coefficient (two dichotomous variables).
Point-biserial is equivalent to the t-test because both test whether a continuous variable differs between two groups; one converts to the other
A team measures three well-being outcomes (life satisfaction, positive mood, flourishing) and gym attendance.
Only flourishing correlates with gym attendance. The published introduction argues from page one that “flourishing is the theoretically relevant outcome,” and the other two measures are never mentioned.
Identify BOTH questionable practices and describe what an open-science version of this paper would look like.
(1) Selective outcome reporting, a form of p-hacking: three DVs were tested and only the significant one was reported.
(2) HARKing: the flourishing rationale was written after the results came in.
An open version would report all three outcomes, label the flourishing result as exploratory, share data and materials, and confirm the finding in a new preregistered sample.
Someone argues: “Punishment always reduces behavior. Spanking is punishment. Therefore, spanking reduces misbehavior.”
Name the method of knowing, explain exactly where it can fail and describe how empiricism would handle the question.
Logic/rationalism
False premise - punishment does not always reduce behavior.
Empiricism would systematically measure misbehavior in children who are and aren’t spanked, for example with a longitudinal correlational study, because experimentally assigning spanking would be unethical.
A sleep scientist asks, “How does circadian timing shift during adolescence?” A school district asks, “Will moving start time from 7:30 to 8:30 reduce first-period tardiness?” Classify each as basic or applied, explain how one informs the other, and write a hypothesis the district could test.
The scientist’s question is basic research (understanding a fundamental process); the district’s is applied (solving a practical problem). Basic findings that teen sleep cycles shift later give the theoretical reason to expect later start times to help. Hypothesis: “Schools that switch to an 8:30 start will show fewer first-period latenesses/ tardiness per week than schools that keep a 7:30 start.”
Identify every problem with this survey item, then how might you rewrite it: “Don’t you agree that professors who assign too much homework and don’t offer extra credit are unfair? Yes / No”
Problems: it’s leading (“Don’t you agree”), double-barreled (homework AND extra credit), vague (“too much”), loaded (“unfair”), and limited to a yes/no response that loses variability. Rewrite as two items, e.g., “How reasonable is the amount of homework in this course? (1 = Very unreasonable to 5 = Very reasonable)” and a separate item about extra credit.
You create a “Climate Anxiety Scale.” Describe the full body of evidence you would gather to show it is valid: at least three types of validity, plus what result would count as support for each.
Validity:
Content (experts confirm the items cover the construct’s facets);
Convergent (moderate-to-strong correlation with general anxiety or eco-worry measures);
Discriminant (weak correlation with unrelated constructs such as social desirability);
Concurrent (correlates with current pro-environmental behavior);
Predictive (predicts future climate activism or distress).
A tightly controlled lab study with 120 U.S. undergraduates finds that a 10-minute gratitude-writing exercise increases helping toward a confederate who drops papers.
Evaluate the study’s internal AND external validity, naming one external validity concern each for subjects, settings and measures.
Internal - assuming random assignment and a comparison condition good internal validity
External
subjects - only undergraduates
settings - tightly controlled lab may not be translated to real world
Measures - is helping only picking up papers? Would this translate to other helping behaviors?
You’re surveying employees about workplace drug use and need to re-survey the SAME people 6 months later. Can you promise anonymity, confidentiality or both?
True anonymity is hard because responses must be linked across time, so the promise is confidentiality. You can get close to anonymity with self-generated ID codes that link responses without identities.
bonus: store any key separately and encrypted, report only aggregate data, never share data with the employer, and consider a Certificate of Confidentiality. The consent form must cover purpose, procedures, risks (including legal and job risks), the limits of confidentiality, that participation is voluntary, the right to withdraw without penalty, and contact information of the researchers.
A researcher finds r = −.45, r² = .20, between weekly volunteering hours and depression scores (n = 200, p < .001).
Interpret the strength and direction, explain r² describe how to plot the data and what to check in the plot, and give two reasons the researcher can’t conclude that volunteering reduces depression.
moderate negative correlation
more volunteering goes with lower depression.
r² = .20, so about 20% of the variance in depression is shared with volunteering
Plot as a scatterplot and check that the relationship is linear, look for outliers and variance spread
Also plot if values for each variable are normally distributed
Can’t infer causation because of directionality (depression may cause people to volunteer less) and third variables (free time, income, physical health or social support could drive both).