Measurement Approaches
Open-ended & closed-ended questions are two broad types of self-report questions. OPEN-ENDED questions are free, unconstrained responses. CLOSED-ENDED questions are when you choose from specific, present options. Name at least one advantage and one disadvantage for each type. ***25 pts each
OPEN-ENDED:
Advantage- more detailed, "true" response.
Disadvantages- take long time to complete content analysis, responses may be unclear, may get irrelevant data
CLOSED-ENDED:
Advantage- easy to score, no intense coding necessary.
Disadvantages- answering biases, limited response, less specific/true response, harder to create, less detail & complexity of response
a. This form of validity is when the operational definition captures all parts of the theory/concept being studied. This is a subjective assessment.**25pts
b. This form of validity is when the measure generally appears to be an accurate measure of the construct. This is a subjective assessment and is the initial judgment. **25pts
c. Criterion validity is when a measure is related to ____-_____ outcomes.**25pts
d. What is the difference between predictive and concurrent validity? **25pts
1) CONTENT VALDITIY
2) FACE VALIDITY
3) REAL-WORLD
4) PREDICTIVE VALIDITY- measure is consistent with criteria outcomes in the future
CONCURRENT VALIDITY- measure is consistent with criteria outcomes in the present
For 25pts each...
1) How well a sample represents the population impacts __________ validity and depends on sampling technique.
2) Probability sampling is generally __________ (less representative/more representative) of the population.
3) Biased sampling is generally __________ (less representative/more representative) of the population.
4) READ CAREFULLY: Margin of error _____ (can/cannot) be calculated for biased sampling. Margin of error _____ (can/cannot) be calculated for probability sampling.
1) EXTERNAL
2) Probability sampling is generally MORE REPRESENTATIVE of the population.
3) Biased sampling is generally LESS REPRESENTATIVE of the population.
4) Margin of error CANNOT be calculated for biased sampling. Margin of error CAN be calculated for probability sampling.
Fill in the blanks. 20 PTS EACH.
1) _______ is the variable we want to predict known as the outcome variable
2) _______ is the variable that precedes/we believe comes 1st
3) A _________ _______ indicates how well the predictor variable predicts the criterion while controlling for other predictors.
4) 1 increase in predictor unit leads a change in X amount of criterion unit means the regression coefficient is ___________ (unstandardized/standardized) -easy to see how scores change in real-world units
5) A 1 SD increase of predictor leads to a change in X amount of criterion SDs means the regression coefficient is ______________ (standardized/unstandardized) -easy to compare relative strengths of predictor variables
1) CRITERION
2) PREDICTOR
3) REGRESSION COEFFICIENT (b or Beta)
4) Unstandardized (b)
5) Standardized (Beta)
Samples can sometimes STILL not be representative in probability sampling due to nonresponse. Why is that the case? How can help with representatives by decreasing non-response (i.e., get people to participate)?
Some people just not interested enough and/or don't have time. Offer incentives, make it accessible, motivate, etc.
Define these examples of CLOSED-ENDED questions.
multiple-choice **66 pts
Likert scale **67 pts
Bipolar adjective **67 pts
MULTIPLE-CHOICE- participants choose the best out of options
LIKERT SCALE- participants select a level of agreement (from strongly disagree --> strongly agree)
BIPOLAR ADJECTIVE SCALE- participants select a level on a scale with two opposing anchors (very unhappy --> very happy)
What type of reliability is being described? **100pts
a. You measure the same individuals twice over time and correlate their scores to themselves (participant A's score vs. Participant A's score). Strong positive correlations around r>_.7 indicate high reliability. This form of reliability answers the question, "Does the test produce same scores with the same circumstances?"
b.This form of reliability is critical when responses need to be coded and often includes multiple observers. This type of reliability can be found by calculating the percent of times raters agreed (good) or plotting each rater's score and calculating the correlation (better) Answers the question, "Is the operational definition clear enough that different people will come to came conclusions?"
a. TEST-RETEST RELIABILITY
b. INTERRATER RELIABILITY
For 100 pts each...
1) Explain what simple-random sampling is (aka most effective method to minimize sampling error).
2) Explain how systematic sampling is used.
3) For 25 BONUS PTS, "Generally, probability sampling is mostly necessary for ____________ research. Biased sampling is typically sufficient & economical for ___________ & _________ research.
1) SIMPLE-RANDOM- Researcher takes list of all members of the population known as SAMPLING FRAME, then participants are randomly selected from entire list (ex: random number generator)
2) SYSTEMATIC- participants chosen using a random starting point & non-random algorithm (ex: two random numbers selected using cpu like 3 & 8. Researcher starts with 3rd person and every 8th person from there gets selected) -can be used when you don't have population in advance like ER ex., & it's probability sample bc we can calculate likelihood each person had of being in sample
3) Generally, probability sampling is mostly necessary for DESCRIPTIVE research. Biased sampling is typically sufficient & economical for CORRELATIONAL & EXPERIMENTAL research.
Answer each question for 50 pts!!
1) What type of longitudinal design tests the relationship between 2 variables by simultaneously studying different groups of people?
2) What type of design looks at groups of participants over time and allows within & between-person comparisons and does not prevent cohort/attrition effects, but can identify when they occur?
3) What type of deign follows a particular person/group over a long period of time to test age effects?
4) Longitudinal designs help us to establish ________ ___________
** 25 pts BONUS: Cross-lag correlations measure 3 types of correlations: cross-sectional, auto, and cross-lag correlations. ________ assesses whether earlier variable measure correlates with later variable measure (type of correlation)
1) CROSS-SECTIONAL DESIGN
2) CROSS-LAG DESIGNS
3) LONGITUDINAL DESIGN
4) TEMPORAL PRECEDENCE
*****Autocorrelations
100 pts each...
_________ _________ is a potential issue in correlation when a smaller more homogenous subset is used that then can misrepresent the true overall relationship.
^ What is the solution here?
1) RESTRICTED RANGE
2) sample from the full range on each variable
Participants' survey responses from be affected by the way we choose to word questions. Some include: unclear questions (participants respond to THEIR understanding), leading questions (suggest a particular answer), and double-barreled questions (asks two things). Identify the type of question each example would fall under for 100 pts each.
1) "Do you think college students should be required to take history and business classes?"
2)"Are you healthy?"
3) "What problems do you have with your significant other?"
1) "Do you think college students should be required to take history and business classes?" >DOUBLE-BARRELED
2) "Are you healthy?" >UNCLEAR
3) "What problems do you have with your significant other?" >LEADING
There are 3 ways to measure INTERNAL RELIABILITY (i.e., make sure each item is assessing the same construct) that we talked about in class. *100 pts each
__________ -correlate each item to the scale total (without that item) (r>_.30)
___________ -split test in half, then correlate scores for halves (r>_.70)
_____________ -compute average correlation among all items in test (typically best way to assess internal reliability w/ large surveys)
ITEM-TOTAL RELIABILITY -correlate each item to the scale total (without that item) (r>_.30)
SPLIT-HALF RELIABILITY -split test in half, then correlate scores for halves (r>_.70)
CRONBACH'S ALPHA -compute average correlation among all items in test (typically best way to assess internal reliability w/ large surveys)
What are the 3 main things error of estimation (sampling error) is based on? **100pts per correct answer
**25 BONUS PTS: A sample of a size that provides a reasonable estimate at reasonable cost and effort is called a _________ _______
SAMPLE SIZE- As sample size increases, the better the population will be represented (small sample is fine when there is little variability, bigger sample needed to account for larger variability)
POPULATION SIZE- The smaller the population (original group), the better a given sample size of that population will represent it. (ex: pulling from SU over UT bc of smaller population right off the bat)
VARIANCE- the more variance, the more difficult it is to estimate population values (the bigger the variance, the bigger the margin of error will be)
****ECONOMICAL SAMPLE
Fill in the blanks. 100 pts each!
1) ________ are external variables that change the nature of the relationship between the variables and can be clarifying explanations (ie., looking at a relationship of two variables with a change in context such as looking at the relationship between two variables and seeing if only looking at females/males changes anything)
2) ___________ are external variables that provide alternative explanations for the relationship between variables
3) _____________ are internal variables that explain the mechanism behind the relationship between variables
1) MODERATORS
2) THIRD-VARIABLES
3) MEDIATORS (A is related to B, bc A is related to C which is related to B, [A-->C--->B])
In a multiple regression analysis, how do you determine which predictor variable is the best predictor for the criterion (outcome variable)?
Look at Betas, then determine which are significant. The best predictor is the one whose Beta is furthest away from 0.
2 ways participants can decrease the validity of responses include response sets & social desirability responding.
1) For 50 pts each, what are two examples of response sets? **hint** RESPONSE SETS are when participants provide responses in a particular pattern. 50 pts per solution for each example.
2) For 100 pts, define social-desirability responding. 100 pts for a solution.
1) a. 'YEA-SAYING'/ACQUIESCENCE vs. 'NAY-SAYING' (solution ex: adding reverse-scored items)
b. 'FENCE-SITTING' (solution ex: remove midpoint/include "N/A" in middle of scale)
2) SOCIAL-DESIRABILITY RESPONDING- when participants respond in a way they think will make them appear a certain way (commonly to "look good") (solution ex: make anonymous, encourage honesty, use bogus pipeline, add filler questions to disguise purpose of study)
Criterion validity can be evaluated with a ______-_____ _______. **hint** researchers see whether measure identifies differences between groups whose behavior SHOULD differ.
KNOWN-GROUPS PARADIGM (ex: on physiological measure results on lie detector test, heart rate and skin conductance levels should differ between lies and truth)
For 100 pts each...
1) Convenience sampling, quota sampling, and purposive sampling are all forms of _____ -samples
2) This type of sampling is similar to stratified random sampling in that researchers identify subcategories of a population of interest, then attempt to fill targeted percentages or numbers for each subcategory. (ex: 90 Asian Americans, 90 African Americans, etc.) In this type of sampling, participants are selected non-randomly.
3) A sample of participants who are readily available and is an easy way to sample. Involves self-selection in which participants take an interest in the outcome, and therefore, are likely to be systematically different.
4) This type of sampling intentionally targets a specific group of people and is frequently used when a specific sample of the population is necessary to represent the population because the research question relies on very specific types of people. (ex: researchers studying an intervention technique to stop smoking would need smokers for study)
**For 25 BONUS PTS, ______ sampling is a form of purposive sampling where participants themselves recruit or recommend other individuals for the study (such as SU students sharing your survey with their friends)
1) BIASED
2) QUOTA SAMPLING *get them to come to you!(researcher samples from population non-randomly until quotas are filled -such as using convenience or purposive sampling)
3) CONVENIENCE SAMPLING
4) PURPOSIVE SAMPLING
** SNOWBALL SAMPLING
For 200 pts each,
1) Explain what problem can longitudinal designs have that we talked about in class.
2) Explain what problem cross-sectional designs can have that we talked about in class.
1) SELECTIVE ATTRITION- may not have same group over time due to non-random dropout (possible that ppl healthier and better educated tend to stay longer)
2) COHORT EFFECTS- may not always be based on generation, bc people born at different times experience different influences, age & cohort always confounded
For effect size, we can use coefficient of determination known as r(squared). This is the percentage of variability accounted for by other variable.
*Pretend there is a positive correlation between happiness and the amount of time a person spends with their dog. r = .47, and r^2 = .22. State how you would describe the effect size in a sentence.
22% of the variance in happiness can be predicted by the amount of time a person spends with their dog.
The amount of time a person spends with their dog explained 22% of the variability in happiness levels.
1) Naturalistic Observation is observing behavior in a natural environment without intervention. Contrived Observation is observing behavior in a lab setting. These forms of observation can be disguised or undisguised. Name 3 solutions (150 pts each) to account for observational issues that we talked about in class.
2) ________ _________ is when observers record what they expect to see rather than what actually occurred (50 pts)
1) PARTIAL CONCEALMENT- participants know they are being observed, but not the behaviors being observed (decreases reactivity and avoids ethical issues)
2) KNOWLEDGEABLE INFORMANTS- behavior reported by other people who know the participant well (may not have to worry about demand characteristics)
3) UNOBTRUSIVE MEASURES- using behavioral artifacts/"signs" of the behavior rather than observing behavior directly (eliminates reactivity but may not always be reliable)
(ex of things to account for: disguised observation can violate informed consent, undisguised can influence demand characteristics & reactivity)
Define Convergent validity VS. Discriminant validity. **250 pts each
CONVERGENT VALIDITY- determines if the measure correlates with other instruments that measure similar conceptual variables (we want a strong relationship between like things)
DISCRIMINANT VALIDITY- determines if measure correlates with other instruments that measure dissimilar conceptual variables (we do NOT want a correlation between instruments that measure dissimilar variables/thinks not alike)
For 125 pts each,
**for 25 BONUS PTS, simple random sampling, stratified random sampling, cluster sampling, & systematic sampling are all types of ________ sampling
1) This type of sampling is when you randomly select naturally-occuring clusters for a sample when people are already divided into arbitrary groups. Specifically, randomly pick clusters and take all members in each cluster. (ex: you randomly select 59 out of 200 high schools [clusters] in a given state and take all students from each high school)
2) This is a variation of stratified random sampling in which researchers intentionally overrepresents a group, where a target demographic creates a disproportionate fraction of sample. In this type of sampling, data is weighted by population proportions so researchers don't over/underestimate responses by not being completely representative.
3) This derives from cluster sampling, where clusters are randomly selected, then participants are randomly selected within each cluster (ex: you randomly select 59 out of 200 high schools [clusters] in a given state, THEN randomly select from the students at each chosen high school)
4) This type of sampling is when the population is divided into strata and then participants are randomly selected within each stratum (i.e., randomly sample from each subgroup & make sure % proportion of sample matches the % proportion in population)
**for 25 BONUS PTS, simple random sampling, stratified random sampling, cluster sampling, & systematic sampling are all types of ________ sampling
1) CLUSTER SAMPLING
2) OVERSAMPLING
3) MULTISTAGE SAMPLING
4) STRATIFIED RANDOM SAMPLING
***PROBABILITY
In multivariate designs (which help us determine internal validity), mediation models allow us to test predictions about what could be linking two variables together and calculate the relationship with other variables statistically removed.
Explain the steps when testing mediation. (Hint list the 4 steps we talked about in class)
1) Establish correlation C, (see if significant) between predictor & criterion (aka relationship of interest).
2) Establish correlation A, between predictor and mediator (see if significant)
3) Establish correlation B, between mediator and criterion (see if significant)
4) Establish correlation C', between predictor and criterion (aka relationship of interest again, accounting for mediator) (see if significant and change in value)
***looking @ C & C', if both significant and close in value, NOT mediated. If significant, but far away from each other in value, PARTIALLY mediated. If not significant, FULLY mediated.
Larger effect sizes are more likely to be significant.
TRUE OR FALSE?
Increasing N decreases the likelihood that a small effect will be significant.
FALSE!! Increasing N INCREASES the likelihood that a small effect will be significant. (if you increase sample size, you are more likely to get effect size)