Bias Detective: Question Wording
A questionnaire begins with "How often do you use AI?" What's a potential issue?
Vague (doesn't specify timeframe, use case, what "AI" means)
A survey asks: “In the past 14 days, how often did your computer crash?
[ ] More than once a day
[ ] Daily
[ ] Once a week
[ ] Less than once a week”
What problem does this have?
Missing intervals (what about twice in a week?)
You’re testing whether dark mode affects reading speed. What type of variable is dark/light mode?
Independent variable
RQ: How common is AI chatbot use for coursework among Brown students, and does it differ by year?
What method would you use?
Survey - descriptive question across a population
Many crosswalk buttons don’t actually impact traffic lights, yet people press them. In design terms, what’s mismatched?
Perceived vs. actual affordances (button signifies a false affordance)
A questionnaire asks “How satisfied do you think your classmates are with Brown’s dining halls?” What problem does this question have?
Asking respondents to report another person's opinions or feelings
A questionnaire asks: “How often do you exercise?
[ ] Regularly
[ ] Occasionally
[ ] Never."
What problem does this have?
Vague wording / vague response categories
You want to compare two versions of an app by having one group of participants test Version A and a different group test Version B. What type of experimental design is this?
Between-subjects / between-group design
RQ: How do content moderators understand community norms and make moderation decisions in practice?
What method would you use?
Interview - you want to elicit rich experiential knowledge
A study shows that LLMs, when asked to generate fictional stories, often portray characters from non-Western countries using reductive stereotypes. What kind of harm is this?
Representational harm
A survey asks "How many times did you use an Internet search engine over the past 6 months?" What problem does this question have?
Recall bias (accurately recalling a precise count is difficult or impossible)
A questionnaire asks: “Overall, how satisfied or dissatisfied are you with your university’s online course registration system?”
[ ] Very satisfied
[ ] Satisfied
[ ] Somewhat satisfied
[ ] Dissatisfied"
What problem does this have?
Unbalanced scale (three positive levels and only one negative, no neutral)
A company compares productivity between its New York office, which is piloting a four-day workweek, and its Boston office, which keeps a five-day schedule. What type of study is this, and what's one potential issue?
Quasi-experiment (not randomly assigned to two conditions); confounding factors across groups (offices may differ in other ways)
RQ: Does showing job applicants an explanation of why an AI screening tool rejected them change how fair they think the process is?
What method would you use?
Experiment - causal question, measure fairness perception across explanation vs no explanation
Name one reason that low-fidelity prototypes are useful.
fast and cheap, allow for early feedback, feedback can be more open when a design looks less finalized
A questionnaire lists the following statement with a scale from agree or disagree: “Brown should switch to digital or tap ID access for dorms.” What problem does this have?
Double-barreled question (two questions in one)
A survey asks: “How many times a week do you walk your dog?” with response options “0 times,” “1-2 times,” “3-4 times,” and “5 or more times.” What problem does this question have?
Assumption about the respondent (no way to indicate that you don't have a dog)
You design a within-group study where participants first use a default alarm app, and then use a redesigned version. For each, you measure indicators of usability and satisfaction. What's a potential issue with your study design?
Order effects (they always see the apps in the same order) - practice, fatigue, or contrast could favor (or hurt) the redesign
A school district is deciding whether to adopt an AI tutoring tool. They want to understand how parents with different views weigh the tradeoffs and respond to each other’s concerns.
What method should they use?
Focus group - allows for open-ended and collective discussion
A school wants to detect "struggling students". They estimate this by counting how many times a student misses class in a given week. What validity or bias concern does this raise?
Content validity (attendance is a narrow measure of "struggling") or measurement bias (oversimplified proxy)
A questionnaire asks: “How much has the new AI summary feature improved your reading workflow?” What problem does this question have?
Leading question (assumes improvement)
You hand out surveys to AI users with the following question: "Out of the following options, which AI feature do you use most?
[ ] Search
[ ] Summaries
[ ] Chat
[ ] Translation
[ ] Other"
What is a potential issue with this question?
Primacy bias (respondents may be more likely to choose the first answer)
Researchers run a study to test the effect of an AI risk score on bail decisions. Online participants each review a set of simulated defendant profiles; half with risk scores and half without. What is a potential validity issue with the results?
External validity (online participants aren’t judges, profiles are simulated, nothing is at stake --> may not reflect how judges would actually act)
A company rolls out an AI tool that ranks job applicants. They want to understand: (1) How satisfied recruiters across sites are with the tool and (2) How and why recruiters decide whether to override the tool’s rankings.
What method(s) should they use?
surveys to measure attitudes across recruiters + interviews to understand how/why they make decisions with it
Two researchers explain why a ride-hailing app hurts drivers. Analyst A: “Local labor protections are weak.” Analyst B: “The app's opaque payment algorithm creates financial insecurity.” Who is arguing technological politics, and who is arguing social determination?
A: social determination (what matters is the social or economic system the technology is embedded in); B: technological politics (the technology’s own design choices distribute power in particular ways)