Category 1
100
What is A/B testing?
A/B testing is a statistical experiment used to determine whether a change produces a meaningful difference in an outcome.
It involves randomly dividing participants into two groups:
Group A (Control): Receives the original version.
Group B (Treatment): Receives the modified version.
You then compare the results to determine whether the treatment has an effect.
200
How to determine if a difference shown in a/b testing is statistically significant?
Run an appropriate hypothesis test. If p < α (usually 0.05), reject H₀.
300
What does a significance level of 0.05 mean?
A 5% Type I error rate for a valid test under H₀.
400
Why randomly assign customers in an A/B test?
To reduce confounding and support causal conclusions.
500
Which test can compare two group means?
An independent two-sample t-test.
600
Which test compares two conversion rates?
A two-proportion z-test, when its assumptions hold.
700
What does confounding mean?
Confounding occurs when a third variable influences both the treatment (or group assignment) and the outcome, making it difficult to determine whether the treatment actually caused the observed difference.
800
What are the two types of independent two-sample t-tests
900
What is causal inference?
Causal inference is the process of determining whether one variable causes a change in another variable, rather than simply being associated with it.