Sampling
Experimental Design
Bias
Principles of Good Experiments
Chapter 4 Vocab
100
A zoo with 100 male frogs and 200 female frogs wants to test a new food to see if it improves the frog's health. They decide to do an SRS of the frogs using the table of random digits and label the frogs starting at 001 through 300. They choose 50 frogs, starting with the row given below. What are the label numbers of the first 4 frogs chosen?
19385 28176 01833 20100 13727 02114
Frog #193
Frog #201
Frog #001
Frog #114
100
Explain the difference between completely randomized design and randomized block design.
Completely randomized design chooses at random which experimental units will get each treatment. Randomized block design first splits the experimental units into blocks based on similar characteristics that are predicted to affect the response variable. Then you choose at random which units from each block will get each treatment. Then, response variables are compared within their blocks.
100
What is non-response, undercoverage, and response bias?
Non-response: sampled individuals do not respond or cannot be contacted
Undercoverage: sampling frame (group of people you sample from) does not represent everyone in your population
Response bias: survey systematically gets incorrect responses
100
Describe each of the three principles of good experimenting.
1) Control: Control for lurking variables that might affect the response variable. The only systematic differences that should be seen between the groups is the treatment that's administered.
2) Random Assignment: Using chance to assign experimental units to treatments to avoid implicit bias that may occur in a human's assignment.
3) Replication: study must have enough experimental units that differences in treatment groups can be distinguished from random differences between the groups
100
A researcher wants to determine the effect of microwaving vegetables on nutritional value. They take 100 vegetables and cook half of them using a microwave, and the other half in the oven. Then, they measure the vegetables nutritional content.
Is this an observational study or an experiment?
An experiment! They are intentionally influencing the nutritional value of the vegetables.
200
Explain the main similarities and differences between a stratified random sample and a cluster sample.
Both split the population up into groups, but in a stratified random sample, you sample from each group (choose an SRS from each group), and in a cluster sample, you choose an SRS of all the groups (and all individuals in the groups chosen are selected for the sample).
200
Describe what a lurking variable is.
A lurking variable is a variable in the experiment that is NOT an explanatory variable, but influences the response variable in some way. It causes confounding, which is when two variables are associated in a way that makes it impossible to tell which variable is which effect on the response variable. For example, a study shows that drinking coffee is likely to cause lung cancer. BUT, little did the researchers know, most of the sample that said they were coffee drinkers were also smokers, which we know causes lung cancer. So, smoking status was a lurking variable in this study.
200
A survey aims to get NYC students' opinions on the current president, but only private schools agree to participate. So, they perform an SRS of all private schools in NYC.
What type of bias affects the results of this survey?
This is undercoverage -- their sampling frame is just private school students (they only sample from private school students), which does not include public school students, a subset of NYC students.
200
A study of human development showed two types of movies to groups of children. Crackers were available in a bowl, and the investigators compared the number of crackers eaten by children watching the different kinds of movies. One kind of movie was shown at 8 AM (right after the children had breakfast) and another at 11 AM (right before the children had lunch). It was found that during the movie shown at 11 AM, more crackers were eaten than during the movie shown at 8 AM. The investigators concluded that the different types of movies had an effect on appetite.
Which principle of experimental design is violated?
Control! There is a variable (time of day) that confounds the data! This must be kept constant so that we can isolate the effect of movie type of cracker consumption.
200
Define the following:
Experimental unit
Treatment
Explanatory variable
Response Variable
Factor
Experimental unit - the collection of individuals to which treatments are being applied
Treatment - the condition applied to the individual that you are measuring the response to
Explanatory variable - the variable that you change to measure the outcome of
Response Variable - the outcome changed by the explanatory variable
Factor - another word for explanatory variable
300
A college student is putting together a report for her university of making campus more accessible. She decides to survey students, professors, and campus staff (campus security, dining service workers, janitors, etc.) to get their opinion on the matter.
What is her population? How should she obtain her sample? (which sampling method would you recommend)
Her population is the students, professors, and campus staff.
A stratified random sample would make the most sense -- she might use students, professors, and campus staff as her strata and choose an SRS of 50 people from each strata.
300
The government notices a rising number of people with Alzheimer's in Florida. They measure the number of people moving to Florida and the rates of Alzheimer's Disease in Florida and conclude that the two variables are associated.
Is this an observational study or an experiment? Identify a possible lurking variable.
Observational! They are not doing anything to cause the rate of Alzheimer Disease in Florida to go up. One lurking variable is that Florida is a very popular place for older people to retire. Alzheimer's Disease occurs most frequently in older people. Lurking in the background is the unforeseen variable of the age of those moving to Florida!
300
A high school math teacher believes the greater the number of hours of sleep a student gets before taking an exam, the better the score. He interviews the 18 students taking his AP Statistics class the day after the exam and later notes their scores.
a) Is this an observational study or an experiment? b) Comment on any possible sources of bias.
a) observational study -- response variable is not effected by teacher
b) response bias -- may over or under exaggerate their hours of sleep, if they think it'll help their grade
undercoverage -- AP students are not an adequate representation of a whole student population
300
Is a random sample of the population always necessary for a good experiment?
No! The principle of random assignment does NOT require a random sample, just that the group of experimental units is randomly ASSIGNED to treatments! However, if your experimental units are not representative of the WHOLE population, the results cannot be generalized to the whole population -- only generalized to the population that the experimental units are representative of.
300
Identify the experimental units, explanatory variable(s), the treatments, and the response variable(s) in the following experiment:
Researchers want to design an experiment in which tasters will rate color and flavor of fries made from several groups of potatoes: freshly picked, stored for a month at room temperature, or refrigerated for a month and then warmed to warm temperature.
experimental units: the potatoes being made into fries
explanatory variable(s): storing method
the treatments: freshly picked (no storing), stored at room temperature for a month, stored in the fridge for a month and then warmed to room temperature
the response variable(s): the color and the flavor
400
A zoo with 100 male frogs and 200 female frogs wants to test a new food to see if it improves the frog's health. There are 30 different species of frogs, each in their own exhibit. Describe how you would do a cluster sample of the frogs. What is one advantage and one disadvantage to this sampling method?
Answers vary. Should sound something like: The zoo can use the species as clusters and do an SRS of the species, choosing 10 species.
The good thing about this method is that you can just give the whole exhibit the new food, rather than trying to only feed some of the frogs the new food, like you would have to in a SRS or stratified random sample.
The disadvantage is that you might not get the whole picture -- what if some species respond better to the new food than other species?
400
Based on the image below, what kind of an experiment is this? (completely randomized, randomized block, or matched pairs)
This is a randomized block design. Subjects are split up into blocks and then experimented on. The blocks are not pairs of similar experimental units, so it is not a matched pairs design.
400
A campaign manager is curious what percent of voters will and will not vote in an upcoming election. They carry out a phone poll on a random sample of registered voters that includes the question, "Will you stay home and pass up your opportunity to vote in the upcoming election?" Only 5% of the 250 people reached in the poll say they plan on staying home.
What type of bias is present? What is the likely direction of the bias?
This is response bias -- the way in which the question was asked favors one response over another.
The results of the campaign manager's survey will probably underestimate the true proportion of people who plan on staying home, since the question asked favors "no, I won't stay home".
400
Describe an experiment that violates the principle of random assignment.
Answers will vary. One possible answer: Suppose you have all of your experimental units in a room. All the women will receive treatment A, and all the men will receive treatment B.
400
Explain what a matched pairs experimental design in. Be sure to mention why someone would want to use matched pairs design.
The matched pairs design is a special type of the randomized blocked design in experimental design. It has only 2 treatments, and the researchers divide the n experimental units into n/2 pairs. Each pair should be similar (that's why it's a specialized case of randomized block design). Within each pair (i.e. each block), the experimental units are randomly assigned to the 2 treatment groups (for example, by a coin flip) and experimented on.
This design emphasizes control because it reduces the chance of an accidental bias that might occur with a completely random selection from a population by making sure that we are comparing two directly comparable units, since each pair is similar.
500
A zoo with 100 male frogs and 200 female frogs wants to test a new food to see if it improves the frog's health. They select 10 male frogs and 20 female frogs, so that the male frogs each have a 1 in 10 chance of being selected. Is this an SRS? Why or why not?
No! This is not an SRS! Every group of 20 frogs (the total number of frogs being sampled) does NOT have the same chance of being selected. For example, a group of 20 female frogs would NOT be selected, as only groups of 10 male and 10 female frogs can be selected. In this case, a SRS is not the best sampling method.
500
Come up with an experiment that contains a lurking variable. What is the lurking variable? Why is it a lurking variable?
Answers may vary. One possible answer: Men and women are given a math test and told that this is to see which gender is better at math, and on the test, all the men perform better. The researchers conclude that men are better at math. There is a lurking variable here -- the result of stereotype threat! It is the effects of the social stigma that women are bad at math -- stereotype threat occurs when someone is a situation in which they feel like they're at risk of conforming to a stereotype about their social group, and end up confirming that stigma because of an inability to perform due to that threat. This is a lurking variable because the effect on the response variable seems like it is the result of the explanatory variable -- gender -- but is actually the result of a third variable -- stereotype threat presence.
500
Give an example of a survey that could result in response-bias.
Answers will vary. One possible answer: A researcher wants to gauge interest in vegetarianism. They ask in a survey "Are you interested in learning more about how to stop the needless death of living things?"
500
Describe an experiment that would violate the principle of control.
Answers will vary. One possible answer: In a study that shows that drinking coffee is likely to cause lung cancer, researchers forgot to ask about smoking status. As it turns out, most of the sample that said they were coffee drinkers were also smokers, which we know causes lung cancer. So, the researchers did not control for smoking status, which they should have known to!
500
Draw an experiment flowchart for the following experiment:
A school with 220 juniors wants to measure the effect of reading habits on SAT reading scores. They decide to give half the juniors a period where they must read silently every day, and the other half is not given a reading period. They then collect their SAT reading scores in senior year and compare them.
