Reliability
What does it mean to say a scale is reliable?
It is consistent. It means your findings can be repeated (i.e., replicated).
What does it mean to say a scale is valid?
It is accurate. It measures what it claims to measure.
What is a composite scale?
A combination of two or more individual measures in a single measure that results in a single score.
What is a z-score?
A z-score is a standardized score.
• It tells us how far a score is from the mean in standard deviation units.
What is a null hypothesis?
The null hypothesis is the hypothesis of randomness and no difference/no correlation.
What statistic is used to calculate the reliability of a scale?
Cronbach’s alpha assesses whether the items hang together
• Measure of the consistency of items, in terms of their ability to generate similar responses from participants.
What is criterion validity?
When a scale matches other established measures of the same concept.
• Example: A short form satisfaction measure should produce similar results as traditional measures.
Why use composite measures?
Composite measures allow researchers to measure a concept that is too complex to be measured simply, or with one item.
What is the formula for computing a z-score?
Z-scores are computed by subtracting the mean from a score and dividing by the standard deviation.
What is an alternative (or research) hypothesis?
The alternative hypothesis is the prediction that the researcher is trying to test.
‼️ Note: It says that there is a difference and a pattern that cannot be attributed to randomness.
True or False: Your measure can be reliable without being valid.
✅ True. Measures can be consistently inaccurate.
**Please note: This is why it is important to use previously validated scales unless you have plans to validate them yourself.
What is discriminant/divergent validity?
When a scale is different from other scales it should not be similar to.
Example: A measure of Positive Workplace should be different from Workplace Bullying.
True or False: We do not need to calculate reliability for previously validated measures.
❌ False ❌
What is the mean and standard deviation of a distribution of standardized scores (i.e., in a standard normal distribution)?
The M (μ) is equal to 0 and the SD (σ) is equal to 1.
What is the difference between a directional and non-directional hypothesis?
Directional requires a 1-tailed test and non-directional requires a 2-tailed test.
• Note: You either specify a direction for the effect or you don’t (i.e., there will be an association vs. there will be a positive/negative association.
True or False: Your measure can be valid without being reliable.
❌ False
If your measure is accurate, it will also be consistent.
What is face validity?
When a scale looks like it is measuring what it is supposed to.
• It is a judgement call.
True or False: Once you have chosen a validated composite measure, it is fine to pick and choose which items you want to use.
❌ False ❌
What do z-scores allow us to do?
Standardized scores allow us to compare two (or more) scores that are on different scales.
• Example: two COM308 courses just took Exam #2. Class 1's scores ranged from 80 - 98.5. Class 2's scores ranged from 74 - 99.
• Explanation: If we standardize scores from each course, we are putting each class on the same metric (i.e., a standard normal distribution) and able to interpret how students performed in each class.
What does an p < 0.05 mean?
🔍 The largest probability level one can have and still claim significance. It means that a statistic must occur, at random, fewer than 5 out of 100 times (i.e., 5% margin of error).
⚠️Caveat: If p = 0.05, the result is not significant.
p > 0.05 ❌ significant
p < 0.05 ✅ significant
True or False: You should calculate Cronbach’s α BEFORE reverse coding your items.
❌ False
You need to reverse them first.
What is convergent validity?
When a scale is similar to other scales it should be theoretically similar to.
• Example #1: A measure of Disclosure Frequency should be similar to a measure of Disclosure Depth
• Example #2: A measure of Student Motivation should be similar to Affective Learning or Growth Mindset
What are the steps for creating a composite scale?
1.) Reverse code items as needed.
2.) Calculate reliability (i.e., using reverse-coded items).
3.) If reliability is good, average your items together to create a composite scale.
Can z-scores be used to identify outliers? How?
Scores more than +/- 2 SDs from the mean are outliers.
What is an α (or alpha) level?
The largest probability level that one can have and still claim to have a statistical significant result. If the probability associated with a statistic is less than the alpha level, it is considered statistically significant.