Chapter 3
What are the 4 validities? Define at least two of them!
Construct Validity- How well the operational variable reflects the conceptual variable
External Validity - How well the findings of a study generalize to a larger population
Statistical Validity - How well the data analysis supports the claims
Internal Validity - When studying relationships between variables, how well confounds are eliminated
Name one historical example of problematic ethics in research.
Tuskegee Syphilis Study
Milgram Obedience Study
Stanford Prison Experiment
The “Monster” Study (including children’s stuttering)
Explain the difference between measurement validity and measurement reliability.
Measurement Reliability - In research, reliability in measurement refers to the consistency of measurement results.
Measurement Validity - How well the variables in a study are measured ... How well the operational variable reflects the conceptual variable.
Give an example of a double-barreled question or a negatively-worded question!
Double-barreled question – The question is really two questions in disguise
-- Eg. “Do you agree that education is crucially important and that the core values of American society are better taught in private schools than public?”
-- If people have to answer this double-barreled question with a singular answer, it’s harder to reflect what they think about this issue as a whole.
-- There’s no way to reflect their perspective about each individual question
Negatively-worded question – The question introduces double negatives that can confuse
-- Eg. “People who do not drive with a suspended license should never be punished”
What is a masked design?
Masked design - In these studies, observers are not researchers themselves – they aren’t informed of the aims of the study or the expected findings
What are the 3 different types of claims? Define at least one of them.
Association Claims - arguments that two variables are related to one another, or that one variable can predict another
Frequency Claims - refer to just one variable and make an assertion about the levels of that variable
Causal Claims - The levels of one variable cause the levels of the other
What are the 3 principles of ethics in Animal Research? (Hint: Alliteration is key here)
Replacement – Find alternatives to animal research when possible
Refinement – Modify procedures to minimize animal distress
Reduction – Fewest animal participants possible
Even though having more participants increases statistical power, we shouldn’t necessarily do this for animal participants
What is 1 of the 2 ways that measurement validity can be assessed subjectively?
Face validity - Does the measure seem to make sense at face value?
-- Good example: “head circumference” for how well a hat fits
-- Bad example: “head circumference” for intelligence
Content validity - Does the measure cover all/many aspects of the concept or theory?
-- Better example: a test on mathematical, verbal, problem solving, and pattern recognition behaviors for intelligence
-- Worse example: a test on only mathematical behavior for intelligence
Explain the difference between likert scales and semantic differential scales!
Likert Scale - Participants are provided a prompt or a question and a scale of ordered responses to choose from.
Semantic Differential Questions - Participants are provided a scale that splits two descriptors and indicate the degree of “closeness” to either side
What is reactivity?
Reactivity - Refers to situations in which participant behavior is impacted by the mere presence of someone observing.
Explain the difference between a variable and a constant.
Variable
- Any factor in a study that can have multiple (two or more) possible levels (options, choices, or levels).
- A variable can vary
Constant
- A factor in a study with only one value/level. It does not vary.
What are the 3 core principles listed in the Belmont Report?
Name the 3 types of measurement reliability. Define at least 2 of them!
Test-retest reliability – Are scores consistent each time a measure is applied?
-- Eg. If an IQ test is administered twice in one semester, are the scores consistent?
Interrater reliability – If multiple researchers are observing behavior, do they report it consistently?
-- Eg. Two researchers observe a problem-solving study. A child solves a puzzle in a certain number of steps. Do both researchers count the same number of steps?
Internal Reliability (not Internal Validity) – Is a participant consistent with themselves?
-- Eg. Related self-report questions
----> “How satisfied are you with your job?”
----> “To what extent do you experience job satisfaction?”
Surveys can include open-ended questions and forced choice questions. Define the pros and cons of each!
Open-ended questions - Participants answer however they like
-- Pros - Answers are rich and diverse
-- Cons - Can be very difficult to code and categorize responses (too many different answers)
----> Example: “How do you feel about this lecture?”
Forced choice questions - Participants choose one of the available answers.
-- Pros - Manageable to organize and code
-- Cons - Restrictive set of options
----> Example: “Have you enjoyed this lecture? Yes or no”
What are 2 of the ways that we can limit how much biases affect our behavioral observations? We talked about 3 in class!
Create an unobtrusive environment
-- Unobtrusive environment - One-way mirror, one face in the crowd, etc.
Waiting until participants become comfortable to record observations
Measure behavioral result instead of behavior itself
-- The advantage of this is that you eliminate some of that awkwardness from the researcher that can influence how the participant behaves
What are the 3 different ways to define a variable?
Conceptual definition of a variable
What does it refer to theoretically? How is it defined? (a definition/more precise description)
“Sensory intensity of the light brightness in the classroom”
Construct definition of a variable
What is it? What are we studying?
“Brightness of lights in the classroom”
Operational definition of a variable
How is it measured and/or manipulated in the study? How is it operationalized?
Medium – 1,000 lumens
Bright – 1,500 lumens”
Name 3 of the 5 guiding, core ethical principles that the APA has set forth!
Beneficence (and nonmaleficence)
Justice
Respect for people’s rights and dignity
Same three as Belmont Report
Fidelity and Responsibility
Establish relationship of trust, be accountable for professional activities (clinician, teacher, researcher)
Integrity
Strive to be accurate, truthful, and honest
Criterion validity - Evaluates the statistical association of a measure with a relevant behavior outcome (e.g., with correlation).
-- This is more objective in measure.
-- Eg. Someone’s degree of isolation should correlate with your BDI. We can quantitatively measure this.
-- It can be assessed via the known-groups paradigm, such that different [relevant] populations should be reflected differently by specific measures.
----> Eg. Beck Depression Inventory (BDI)
A shortcut means relying on a response set. What are response sets? What are the 2 kinds of response sets that we talked about in class?
Response sets - This occurs when participants respond to numerous items the same way without necessarily thinking it through. A series of answers that follow a trend that are consistent with each other.
-- Acquiescence (affirmative route) - Participants may be biased in favor of choosing the “yes” option more often than they should “or a positive option like “strongly agree”).
-- Fence Sitting (neutral route) - For Likert scales or semantic differential questions, participants may inordinately favor the middle “safe” option (like “Neutral” or “I don’t know”.)
What are the 3 criteria for establishing causality?
Covariance - As A changes, B changes
Temporal Precedence - The study’s method ensures that A comes first in time, before B
Internal Validity - The study’s method ensures that there are no plausible alternative explanations for the change in B; A is the only thing that changed
Instead of being told they had syphilis, what were the infected individuals told they had instead?
Tuskegee Syphilis Study
600 black men were studied over 400 years
400 infected, 200 not
Those infected were told they had “bad blood”, NOT syphilis
Penicillin treatment available in 1940s, denied to study participants
What principles were violated?
Participants were disrespected
Participants were harmed
Participants were a targeted disadvantaged social group
What statistical measure is internal reliability often assessed with?
Internal Reliability is often assessed with Cronbach’s alpha.
-- NOT to be confused with the “alpha” (𝛼) you learned about in your statistics course
Cronbach’s alpha is a combined measure of internal correlation and the number of items.
-- Generally want a value ≥ .80.
Name 2 of the 3 additional self-report survey issues that we talked about in class (Hint: Think "Walter White")!
Social-desirability - Participants may respond in a way they believe paints them in a good light
-- Using the frame of “If I was a good person or the person I wanted to be, how would I answer this question?” instead of “If I am being truthful about who I am, how would I answer this question?”
“Faking bad” - Participants may respond the opposite way, inordinately emphasizing negative answers (answers that are perceived to be negative/socially undesirable by the participant)
Memory flaws - Participants may simply not remember their own behaviors accurately
-- It’s hard to translate what’s in one’s own mind onto paper, especially if it’s on a limited scale